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ABSTRACT 


To  ensure  user  acceptance  and  success  of  technology-enhanced  training  in  an  era 
of  declining  budgets  and  scarce  resources,  civilian  and  military  organizations  must 
exploit  effective  and  efficient  planning  and  evaluation  tools  as  part  of  simulation-based 
training.  The  purpose  of  this  research  is  to  validate  the  hypothesis  that  computer-based 
(automated)  planning  tools  will  improve  training  planning  effectiveness  and  efficiency 
for  simulation-based  team  training  exercises.  A  modified  pretest-posttest  control  group 
experimental  design  was  used  to  determine  the  effectiveness  and  efficiency  of  a 
computer-based  training  planning  model. 

Performance  measures,  as  well  as,  cognitive  mapping  and  clustering  techniques 
aided  the  assessment  of  the  automated  training  planning  tool.  The  treatment  group  of  27 
two-person  teams  used  a  computer-based  planning  tool  to  develop  a  plan  for  conducting 
simulation-based  training,  while  a  control  group  of  25  two-person  teams  used  current 
manual  planning  methods.  Both  groups  then  executed  their  respective  training  plans  in  a 
constructive  simulation  program  called  Janus. 

Results  indicate  that  the  group  which  used  the  computer-based  planning  tool  had 
a  significantly  better  mental  model  of  the  training  planning  process  than  the  control 
group.  Based  on  statistical  tests  comparing  relevant  performance  measures,  the  study 
concludes  that  a  computer-based  planning  tool  has  the  potential  to  enhance  the 
effectiveness  and  efficiency  of  simulation-based  team  training. 
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CHAPTER  1 


INTRODUCTION 

Overview  of  Simulation-Based  Training  Tonis 

Aided  by  the  acceleration  of  computational  and  database  processing  power, 
simulations  are  enhancing  conventional  training  programs.  As  a  result  of  recent  research 
focusing  on  training,  Bouwens,  Jones,  and  Pierce  (1994)  conclude  that  growth  continues 
in  the  number  of  opportunities  for  skill  and  task  training  in  simulation-based 
environments.  Simulations  often  involve  several  persons  cooperating  as  a  team  to  fulfill 
training  objectives  and  tasks. 

With  the  advent  of  simulation-based  training  technologies  for  increasing  training 
preparedness,  industry  and  academia  strive  to  assess  the  training  enhancement  offered  by 
computers  (Schank,  1994).  Improving  productivity  with  technology  continues  to  evolve 
as  a  potential  aspect  of  everyone’s  future  (George,  Malcolm,  &  Jeffers,  1993). 
Consequently,  computer-based  simulations  play  an  important  role  in  civilian  and  military 
traimng  for  normal,  emergency  and  life-threatening  activities.  Examples  are  simulation 
training  for  pilot/astronaut  planned  and  contingency  situations,  nuclear  power  plant 
normal  and  emergency  operations  and  military  combat. 

Using  existing  manual  planning  methods,  simulation  operators  often  assist  those 
responsible  for  training  in  translating  training  requirements  into  executable  training 
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exercises.  Trainees  seldom  understand  how  their  training  requirements  and  plans  are 
developed  nor  do  they  consider  the  need  for  simulation  resource  coordination.  Poor 
planning  and  coordination  often  results  in  a  time-constrained  adjustment  of  training 
causing  negative  impact  on  the  use  of  the  simulation  and  the  team’s  training  proficiency. 

An  extensive  review  of  the  literature  has  not  identified  any  computer-based  tools 
being  used  for  simulation-based  training  planning.  The  reason  that  such  a  tool  may  be 
nonexistent  is  that  training  research  development  efforts  seem  to  have  concentrated  on  the 
simulations  instead  of  the  entire  training  planning  process.  Even  though  the  users  can 
articulate  training  goals  and  objectives  well,  the  linkages  between  the  organization’s 
training  goals  and  simulation-based  training  outcomes  must  be  well-defined  (Keiser  & 
Seeler,  1987).  The  integration  of  the  remaining  elements  of  automation,  training, 
planning  and  evaluation  will  achieve  an  effective  and  efficient  system  for  the  training 
planning  process. 


Need  for  the  Study 

Training  personnel  continually  seek  ways  to  improve  and  maintain  training 
proficiency  of  the  orgamzational  members  and  teams/crews.  Frequently,  the  actual 
training  event  is  difficult  to  replicate  unless  that  event  is  realistically  portrayed  through 
simulation  (Spitzer,  1993).  Simulation-based  training  must  be  conducted  formally  to 
ensure  the  trainees  do  not  adopt  incorrect  behaviors  which  may  cause  ill-fated 
consequences  (Noomahan,  1993). 
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To  enhance  understanding  of  the  training  planning  process,  the  trainer  must 
quickly  become  familiar  with  organization  procedures,  practices,  and  regulations  for 
training.  Using  the  appropriate  references  and  documents,  the  trainer  must  prepare  a 
well-defined  and  organized  training  plan  to  utilize  simulation  resources.  Training 
planning  staffs  often  react  quickly  to  events,  while  being  held  accountable  for  ensuring 
well-planned  and  executed  training  exercises.  The  trainer’s  understanding  of  the 
simulation-based  training  planning  exercise  is  critical,  but  is  dependent  on  experience  and 
learning.  For  current  and  future  training  needs,  a  delay  in  training  planning  skill 
acquisition  is  often  imacceptable. 

The  requirement  to  reduce  costs  is  relevant  in  terms  of  funds  and  time.  The 
reduction  of  training  budgets  must  be  aligned  with  increased  computer  power. 

Generating  training  scenarios  usually  requires  hours  of  manual  planning  and  other  paper¬ 
intensive  activities.  Insufficient  planning  causes  poorly  executed  training  and  wastes  one 
of  an  organization’s  limited  and  valuable  resources —  training  time.  To  prevent  wasted 
resources,  training  program  developers  must  align  the  goals  of  organizational  training 
requirements  with  the  simulation-based  training  process. 

A  trainer  needs  to  have  a  planning  tool  which  automates  the  training  planning 
process  to  develop  training  which  results  in  explicit  feedback  and  linkages  to  future 
planned  exercises.  For  a  computer-based  tool  to  assist  users,  it  must  replicate  a  process 
and  reduce  many  routine  and  non-judgmental  tasks.  Schneiderman  (1992)  insists  that 
users  can  “avoid  the  annoyance  of  handling  routine,  tedious,  and  error-prone  tasks,  and 
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can  concentrate  on  critical  decisions,  planning,  and  coping  with  unexpected  situations” 
(p.  83).  Schneiderman  also  believes  that  computers  should  track  large  amounts  of  data 
and  information  using  preset  patterns  and  procedures.  As  a  result,  planners  have  more 
time  and  energy  to  make  decisions,  plan  and  react  to  crises  (Schneiderman,  1992). 

There  is  a  need  to  improve  the  effectiveness  and  efficiency  for  team  training  plan 
preparation  for  simulation-based  training.  To  ensure  user  acceptance  and  success  of 
simulation-based  training  in  an  era  of  declining  budgets  and  scarce  resources,  civilian  and 
military  orgamzations  must  improve  the  effectiveness  and  efficiency  of  training  planning 
One  of  the  primary  drawbacks  hindering  the  development  of  a  tool  to  satisfy  those 
training  planning  requirements  has  been  the  lack  of  computer  programs  to  automate  tasks 
and  activities.  This  study  will  provide  the  structure  of  a  process  to  examine  these  issues 
for  augmenting  future  training  planning  with  capabilities  for  handling  the  complex 
requirements  encompassing  simulation-based  training 

In  summary,  there  is  a  need  to  investigate  the  value  of  automated  planning  tools 
for  simulation-based  training.  These  tools  are  important  in  that  they  have  the  potential  to: 

1 .  improve  and  maintain  training  proficiency  through  effective  planning, 

2.  enhance  understanding  of  the  training  planning  process  to  accommodate  large 
turnovers  in  training  personnel, 

3.  reduce  costs  to  meet  shrinking  training  budgets  for  simulations  and  support 
personnel,  and 
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4.  automate  the  training  planning  process  to  achieve  better  training  results  and 
build  training  outcome  linkages  to  future  planned  exercises. 

Statement  of  the  Problem 

Simulation  training  plan  development  is  not  being  done  as  effectively  and  as 
efficiently  as  possible  due  to  a  lack  of  understanding  of  the  training  plan  development 
process  and  the  unavailability  of  computer-based  tools.  Planning  for  training  requires 
knowledge  and  experience  in  the  training  planning  development  process.  For  the  most 
part,  manual  methods  (non-computer  based)  are  still  being  used  to  develop  training  plans 
for  classroom,  simulator,  and  insitu  training  environments.  Improvements  are  critically 
needed  in  training  planning  for  simulations  in  order  to  enhance  personnel  training 
through  effective  training  plans,  and  efficient  time,  manpower  and  money  utilization. 

Major  questions  related  to  this  problem  are: 

1 .  What  contributions  do  computer-based  planning  tools  offer  to  simulation- 
based  team  training? 

2.  What  is  the  best  method  of  computer-based  training  planning? 

3.  What  are  some  improved  quantitative  measures  of  training  effectiveness 
required  to  assess  training? 

The  first  question  (What  contributions  do  computer-based  planning  tools  offer  to 
simulation-based  team  training)  is  the  primary  focus  of  this  study.  Some  researchers  and 
practitioners  believe  that  computer-based  tools  contribute  to  productivity,  but  there  are 
few  tests  which  corroborate  this  point.  Very  few  authors  provide  empirical  evidence  or 
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logical  methods  which  support  the  technology  beyond  hearsay  or  subjective  assessment 
(Keiser&  Seeler,  1987). 

According  to  Eberts  (1994),  “the  computer  user  perceives,  stores  and  retrieves 
information”  from  memory,  then  synthesizes  that  information  to  conduct  decisions  for 
problem-solving  and  future  action  (p.  139).  Measured  in  terms  of  the  overall  contribution 
to  the  organization’s  training  proficiency  of  teams  and  individuals,  the  success  of  a 
simulation-based  training  exercise  relies  upon  consistent  and  thorough  planning  The 
training  planning  process  must  examine  the  development  of  scenarios  and  conditions 
specific  to  simulation-based  training.  There  are  existing  manual  methods  which  have 
become  obsolete  due  to  the  enormous  detail  and  planning  involved.  Therefore, 
computers  can  assist  in  routine  processes,  as  well  as,  the  guidance  of  the  users  through  a 
logical  sequential  operation  for  training  plan  development. 

Hypothesis:  The  use  of  a  computer-based  planning  tool  instead  of  current  manual 
methods  will  improve  the  training  plan  development  process  for  a  constructive  simulation 
exercise.  Within  this  environment: 

1 .  Computer-based  plarming  tools  will  enhance  the  mental  model  of  the  training 
plan  development  process. 

2.  More  effective  training  plans  will  be  created  and  results  achieved  with 
computer-based  planning  tools. 

3.  Simulation-based  training  planning  will  be  accomplished  more  efficiently 
using  computer-based  tools. 
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In  order  for  the  reader  to  understand  some  of  the  words  and  phrases  used 
throughout  the  study,  definitions  are  now  presented.  In  addition,  Appendix  A  includes  a 
list  of  the  abbreviations  and  acronyms  used  in  this  study. 

Definition  of  Terms 

Computer-based  tool.  A  device  or  system  based  on  digital  computer  technologies. 

Computer.  A  programmable  (changeable)  device  that  accepts  input,  manipulates 
data  and  outputs  data  in  some  form”  (Shedroff,  Hutto,  &  Fromm,  1993,  p.  134). 

Con?tructive  simulation.  A  simulation  with  a  game  component  which  involves 
human  interaction  and  computer  models  for  replicating  realistic  scenarios. 

Distributed  interactive  simulation  (DTSf  A  simulation  architecture  which  allows 
large  numbers  of  sunulated  systems,  both  manned  and  unmanned,  at  different  locations  to 
interact  at  the  same  time  to  accomplish  a  common  training  mission  via  communication 
networks  (Kazarian  8c  Schultz,  1994). 

Effectiveness.  The  extent  to  which  the  structure  leads  to  improved  performance, 
makes  a  difficult  task  easier,  or  enables  accomplishing  a  task  that  could  not  otherwise  be 
accomplished.”  (Rouse,  1991,  p.  23). 

Efficiency.  The  ratio  of  input  resources  versus  output  results. 

Free  recall.  The  depiction  of  steps  or  components  of  a  process  using  textual  or 
pictorial  representation. 
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(lasie-  “A  structured  activity  in  which  two  or  more  participants  compete  within 
constraints  of  rules  to  achieve  an  objective”  (Keiser  &  Seeler,  1987,  p.  460). 

Manual  tool.  Any  device  which  does  not  use  electronic  means  of  calculation, 
organization  or  data  manipulation. 

Mental  model.  Arrangement  of  information  into  structured  patterns  which  allow 
versatile  storage  and  retrieval  of  information  in  terms  of  features  related  to  situations, 
entities  and  environmental  conditions  (Tannenbaum,  Cannon-Bowers,  Salas,  &  Mathieu, 
1993). 

Position  memory.  The  restatement  of  items  from  memory  recall  which  were 
presented  in  categorical  lists. 

Scenario.  Details  of  the  subject  of  the  simulation,  outlined  with  the  objectives  and 
game  rules  for  the  participants  (Keiser  &  Seeler,  1987). 

Simulation.  “An  operational  model,  using  selected  components,  of  a  real  or 
hypothetical  process,  mechanism,  or  system”  (Keiser  &,  Seeler,  1987,  p.  460);  uses  only 
mathematical  relationships  to  simulate  actions  and  behaviors. 

Simulator.  ‘A  simulation  with  no  game  component  ~  the  use  of  computer-based 
simulators  to  replace  training  on  actual  equipment”  (Keiser  &  Seeler,  1987,  p.  461); 
combines  both  mathematical  and  physical  relationships  for  behavioral  characteristics. 
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Structured  questionnaire.  “A  questionnaire  which  includes  fixed  items,  fixed 


response  categories  and  fixed  sequencing  of  items.”  (Sproull,  1995,  p.  387). 

$_ubiect  matter  expert  tSMEl.  An  individual  who  is  considered  very 
knowledgeable  in  his/her  field  of  study  or  a  specific  area. 

Te^.  “A  distinguishable  set  of  two  or  more  people  who  interact  dynamically, 
interdependently,  and  adaptively  toward  a  common  and  valued  goal/objective/mission, 
who  have  each  been  assigned  specific  roles  or  functions  to  perform,  and  who  have  a 
limited  life  span  of  membership”  (Salas,  Dickinson,  Converse,  &  Tannenbaum,  1992,  p. 
1297). 


Methodology 

The  experimental  design  used  in  the  study  was  the  pretest-posttest  control  group 
design  articulated  by  Gibson,  Ivancevich,  and  Donnelly  (1994).  This  design  is  best  where 
gain  or  change  in  behavior  is  the  desired  dependent  variable.  For  this  study,  the  change 
of  the  user’s  mental  model  of  the  training  planning  process  is  one  of  the  dependent 
variables.  The  other  dependent  variables  are  Task  Load  Index  (TLX)  scores,  performance 
times,  quality  ratings  and  simulation  measures  of  effectiveness  (MOE).  The  independent 
variable  was  the  use  or  nonuse  of  computer-based  planning  tools  to  support  simulation- 
based  training. 

This  study  was  conducted  at  the  University  of  Central  Florida,  Fort  Rnox, 
Kentucky,  and  Fort  Hood,  Texas,  from  May  to  July  1995.  The  sample  population 
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consisted  of  officers,  noncommissioned  officers  and  cadets  who  were  involved  in  the 
training  planning  process.  Officers  normally  have  the  education  for  training  planning, 
while  the  noncommissioned  officers  have  the  experience  in  execution  and  conduct  of 
training.  Cadets  were  used  in  the  pilot  test  in  order  to  have  a  test  population  which  could 
provide  naive  expertise  to  the  evaluation  of  the  experimental  design  and  instruments.  All 
volunteers  were  randomly  assigned  to  either  manual  or  automated  two-person  planning 
teams. 

Prior  to  the  primary  study  data  collection,  a  pilot  test  was  conducted  in  Orlando 
with  University  of  Central  Florida  (UCF)  Army  Reserve  Officers  Training  Corps 
(AROTC)  cadets.  The  pilot  group  assisted  in  identifying  areas  of  concern  through  a 
rehearsal  of  the  experiment. 

Primary  study  data  were  collected  at  Fort  Knox,  Kentucky,  in  May  1995,  and  Fort 
Hood,  Texas,  in  June  1995.  Two-person  planning  teams  consisted  of  an  officer  who  has 
the  training  education  and  a  noncommissioned  officer  who  normally  has  the  training 
experience.  At  Fort  Knox,  the  sample  was  created  by  forming  16  automated  and  15 
manual  two-person  planning  teams  from  an  Armor  battalion.  At  Fort  Hood,  the  study 
sample  consisted  of  1 1  automated  and  10  manual  two-person  planning  teams 
participating  also  from  an  Armor  battalion.  Training  time  was  also  allocated  to  teach  the 
Janus  simulation  and  to  orient  the  computer-based  teams  on  using  the  Training  Exercise 
Development  System  (TREDS)  computer-based  planning  tool.  Analysis  of  the  primary 
study  data  was  conducted  using  nonparametrics  statistics.  The  power  of  the 
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nonparametrics  test  by  Mann- Whitney  was  deemed  more  appropriate  for  the  post-test 
data,  whereas  the  Wilcoxon  signed-ranks  test  was  used  to  compare  the  data  for  the 
pretest-posttest  component  of  the  design  (Conover,  1980). 

Limitations  of  the  Study 

The  limitations  of  the  study  were: 

1 .  A  constructive  simulation  was  used. 

2.  Test  subjects  were  randomly  selected  from  two  military  organizations  versus 
the  entire  population  at  large. 

The  first  limitation  was  that  the  research  focused  on  team  training  using  a 
constructive  simulation.  Constructive  simulations  have  been  used  extensively  for  training 
teams  for  over  ten  years.  Their  recognition  as  viable  training  simulations  is  seen  by  the 
large  annual  funding  and  efforts  applied  to  constructive  simulations’  development.  An 
attempt  to  introduce  a  live  or  virtual  simulation  would  have  greatly  increased  workload, 
scope  and  data  for  the  experiment  and  may  have  provided  less  added  benefit  to  the  study. 
The  limitation  of  using  only  one  type  of  simulation  did  not  appear  to  hinder  gathering 
informative  study  data. 

The  second  limitation  was  that  the  test  subjects  were  randomly  selected  from  two 
military  organizations  versus  the  entire  population  at  large.  The  study  was  limited  to  test 
subjects  randomly  selected  from  two  military  organizations  since  it  was  impractical  to 
test  the  entire  population.  The  sample  population  adequately  represented  the  population. 
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A  demographics  data  form  (in  Appendix  B)  was  developed  to  verify  this  assumption  on 
the  representativeness  of  training  planners  in  the  U.S.  Army. 

The  use  of  a  constructive  simulation  and  designated  units  for  sample  populations 
were  practical  limitations  that  did  not  appear  to  significantly  alter  the  experimental 
conditions  nor  the  results. 

Assumptions 

The  following  assumptions  were  made  concerning  the  conduct  of  the  study: 

1 .  simulation  is  a  valid  method  for  executing  training,  and 

2.  experiences  and  training  knowledge  of  those  tested  were  representative  of 
teams  involved  in  the  Army’s  training  simulation  programs. 

The  first  assumption  for  this  study  is  that  simulation  have  already  been  recognized 
by  trainers  as  a  viable  means  to  improve  team  proficiency  (BCieser  &  Seeler,  1987). 

Studies  have  presumed  that  simulations  do  enhance  team  proficiency  as  the  foundation 
for  future  research  activities  (Gonzalez  &  Ingraham,  1994;  Pemberton,  Classe,  Bradley, 

&  Wilson,  1994).  Verbal  and  written  assessments  obtained  through  interviews  and  the 
post-exercise  questionnaire  (see  form  in  Appendix  C)  later  supported  this  assumption  that 
simulation  improved  training  proficiency. 

The  second  assumption  presupposes  that  the  study’s  subjects  represent  a  cross- 
section  of  the  population  of  simulation-based  training  plaimers.  The  testing  was 
organized  to  test  52  two-person  planning  teams  at  two  military  installations  which  have 
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similar  missions.  Since  the  Army  rotates  leaders  frequently  between  organizations,  the 
assumption  is  that  the  experiences  and  training  knowledge  of  the  groups  tested  represent 
all  teams  involved  in  the  training  simulation  process.  Consequently,  continued  testing 
should  yield  representative  results. 


Summary 

The  advent  of  simulation-based  training  is  proving  to  be  a  mainstay  for  civilian 
and  military  training  in  normal,  emergency,  and  life-threatening  activities.  Areas  of 
application  include  pilot/astronaut  training,  nuclear  power  plant  operations,  and  military 
combat  simulation.  As  various  disciplines  assess  the  importance  of  simulation-based 
training,  issues  surface  regarding  the  preparation  and  optimization  of  this  new  training 
mode. 

The  purpose  of  this  study  was  to  test  the  hypothesis  that  the  use  of  a  computer- 
based  planning  tool  instead  of  current  manual  methods  will  improve  the  training  plan 
development  process  for  a  team  simulation-based  training  exercise.  The  application  or 
nonapplication  of  computer-based  planning  tools  was  the  independent  variable.  The 
methodology  for  this  study  was  a  modified  experimental  pretest-posttest  control  group 
design.  Testing  was  planned  at  two  research  sites  in  Fort  Knox,  Kentucky,  and  Fort 
Hood,  Texas,  for  52  two-person  planning  teams.  The  data  analyses  between  the 
independent  samples  were  conducted  using  descriptive  and  nonparametric  statistics. 
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Limitations  of  the  study  were  that  only  the  Janus  simulation  was  used  and  that  the 
test  subjects  were  only  from  two  organi2ations.  The  first  assumption  was  the  statement 
that  simulation  enables  credible  training  execution.  Secondly,  the  study  plan  assumed 
that  the  experiences  and  training  knowledge  of  the  subjects  adequately  represented  those 
involved  in  military  training  simulations. 

The  original  contribution  of  the  research  conducted  in  this  study  is  the  partial 
validation  of  benefits  derived  from  the  use  of  computer-based  planning  tools  to  enhance 
the  effectiveness  and  efficiency  of  simulation-based  team  training.  Specifically,  the  study 
assesses  whether  computer-based  training  plan  development  for  simulation-based 
training: 

1 .  provides  a  methodology  for  validating  the  use  of  computer-based  planning 
tools  to  enhance  the  effectiveness  and  efficiency  of  simulation-based  training, 

2.  identifies  the  linkage  between  training  task  requirements  and  simulation-based 
training  characteristics  by  building  a  mental  model  of  the  training  plan 
development  process,  and 

3.  assists  future  research  efforts  by  providing  potential  measures  of  effectiveness 
and  efficiency  for  evaluating  technological  advancements. 
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CHAPTER  2 


LITERATURE  REVIEW 

The  purpose  of  this  study  was  to  investigate  the  contributions  that  computer-based 
planning  tools  offer  to  simulation-based  team  training.  This  chapter  presents  the  results 
of  a  literature  review  of  relevant  topics  important  to  answering  the  research  question  and 
identifying  training  metrics.  These  topics  are: 

1 .  the  training  process, 

2.  training  plan  development, 

3.  computer-based  training  planning  tools, 

4.  measures  of  training  effectiveness,  and 

5.  measures  of  training  efficiency. 

These  topics  will  now  be  discussed  in  the  same  order  as  outlined  to  facilitate 
understanding  of  the  entire  study  plan.  Of  primary  importance  in  this  study  is  the  training 
process  as  it  is  influenced  by  all  components  of  this  study.  Secondly,  training  plan 
development  unpacts  on  current  and  future  trainmg  methods  in  both  civilian  and  military 
environments.  Third,  the  availability  and  flexibility  of  computer-based  planning  tools 
was  examined  and  compared  with  manual  planning  methods  for  simulation-based 
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training.  Finally,  the  remaining  two  areas  concerned  performance  evaluation  measures — 
effectiveness  and  efficiency. 

The  Training  Process 

With  recent  trends  toward  simulation-based  training,  understanding  the  systematic 
training  model,  team  versus  individual  training,  and  simulation-based  training  is  critical. 
The  training  model  is  a  representation  of  those  factors  and  training  tasks  which  must  be 
accomplished  to  improve  the  proficiency  of  the  intended  training  audience.  Two  of  the 
most  common  forms  of  training  are  individual  training  and  team  training.  An  overview 
of  simulations  will  serve  to  define  and  highlight  this  media  for  training.  Simulations  also 
include  multiple  and  complex  features  which  plaimers  must  consider  in  training  plan 
development.  The  training  system’s  goal  is  “to  guide  the  development  of  a  cognitive 
model”  (p.  15)  which  will  instill  an  accurate  “representation  of  the  tasks  to  be  performed 
and  the  information  necessary  for  their  performance”  (Williams,  Reynolds,  Carolan, 
Anglin  &  Shrestha,  1989,  p.  15). 

The  Systematic  Training  Model 

In  the  training  process,  the  model  which  typifies  those  actions  and  procedures  for 
training  plarmers  to  follow  is  the  systematic  training  model.  “To  be  systematic,  a  training 
development  process  has  to  be  standardized,  comprehensive,  internally  consistent,  and 
reliable”  (Gropper  &  Ross,  1987,  p.  196).  The  first  representative  systematic  training 
model,  the  Instructional  Systems  Design  (ISD)  model,  was  developed  during  World  War 
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II  by  the  U.S.  military  to  train  soldiers  to  react  to  situations  using  behavioral  psychology 
(Gropper  &  Ross,  1987).  Sloman  (1994)  synthesizes  the  components  of  the  ISD  model 
into  the  five  stages  shown  in  Figure  1 . 


(Sloman,  1994,  p.  40) 

Figure  1.  Systematic  Training  Model 
To  determine  the  training  need,  an  analysis  of  the  training  audience’s 
requirements  forms  the  basis  for  ensuring  that  the  appropriate  training  curricula  is 
designed  and  developed.  Training  execution  or  delivery  is  the  conduct  of  the  planned 
traimng,  while  evaluation  of  the  training  audience  provides  diagnostics  for  the  training 
Sloman  (1994)  asserts  that  future  training  models  need  to  “retain  the  best  elements  of  the 
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systematic  training  model”  and  stipulates  that  the  five  ISD  components  must  be 


augmented  to: 


1 .  offer  the  trainer  a  structured  and  disciplined  fi-amework  in  which  to  work, 


2.  ensure  that  an  effective  loop  from  evaluation  is  in  operation,  and 

3.  emphasize  the  importance  of  quantifiable  results,  even  though  the  feasibility 
of  this  [quantifying  the  results]  varies  (p.  40). 

The  systematic  training  model  has  proven  itself  through  longevity  and  utility. 

These  additions  promise  to  enhance  the  training  model  and  allow  a  developer  to  build  a 

consistent,  effective  and  measurable  training  program.  Organizational  characteristics  and 


training  environments  are  also  important  in  developing  a  training  model.  Consequently, 
Sloman  (1994)  specifies  that  the  effective  training  model  must  also  consider: 


1 .  the  need  to  embed  trainmg  activity  firmly  in  the  organization,  thus  securing 

Imks  to  strategic  objectives  and  permitting  training  to  operate  in  a  corporate 
context,  ^ 

2.  that  the  traimng  function  has  a  role  in  articulating  training  needs  as  well  as 
reactmg  to  them,  and 

3 .  that  different  organizations  are  at  different  stages  in  terms  of  training 
sophisUcation  and  different  approaches  may  therefore  be  necessary  (p.  40). 

In  addition  to  identifying  the  characteristics  and  environmental  considerations  for 

traimng,  the  identification  of  training  needs  in  a  training  model  must  first  determine 

whether  the  training  will  be  for  individuals  or  teams. 
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Individual  Versus  Team  Training 


According  to  Salas,  Dickinson,  Converse,  and  Tannenbaum  (1992),  individual 
team  members  should  be  allowed  time  to  master  individual  skills  in  order  to  achieve 
more  effective  and  efficient  team  training  results.  Individual  simulation-based  training 
often  involves  hands-on  training  with  simulators,  while  team  training  incorporates  a 
collection  of  individuals  who  coordinate  and  conduct  their  tasks  to  accomplish  a  team- 
centered  objective.  When  both  individual  and  team  skills  are  required,  the  optimal  mix  of 
individual  and  team  training  must  consider  the  amount  and  timing  for  each  set  of  training 
to  occur  (Salas,  Dickinson,  Converse,  &  Tannenbaum,  1992). 

The  transition  from  individual  to  team  training  does  not  occur  by  the  creation  of 
the  team  as  a  set  of  two  or  more  individuals.  “Training  for  battle  group  team  members 
usually  progresses  from  sunulation-based  instruction  on  individual  operator  tasks, 
through  simulation  training  for  subteams  and  single-platform  teams,  to  simulation  for 
multiple-platform  teams”  (Tumage,  Houser,  &  Hofmann,  1989,  p.  A-1).  Team  members 
normally  apply  simulation-based  training  to  progress  from  individual  tasks  to  team  tasks. 

Recent  efforts  by  the  U.S.  Navy’s  Naval  Air  Warfare  Center  Training  Systems 
Division  (NAWCTSD)  for  enhancing  training  in  the  U.S.  Navy  encompasses  several 
areas.  The  Navy  s  primary  emphasis  to  convert  training  theory  into  techniques  (Cannon- 
Bowers,  Tannenbaum,  Salas,  &  Converse,  1991)  is  evidenced  by  numerous  applications 
to  team  training  (Hall,  Dwyer,  Cannon-Bowers,  Salas  &  Volpe,  1 993 ;  Tannenbaum, 
Caimon-Bowers,  Salas,  &  Mathieu,  1993;  Rouse,  Cannon-Bowers,  &  Salas,  1992), 
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computer-based  exercise  preparation  (Pemberton,  Classe,  Bradley,  &  Wilson,  1994),  and 
intelligent  embedded  trainers  (Gluckman  &  Willis,  1994;  Cannon-Bowers,  Salas, 
Duncan,  &  Halley,  1994).  Each  of  these  areas  are  based  on  the  Navy’s  team  training 
concepts  and  principles  which  are  divided  into  the  following  levels: 

1 .  preteam  indoctrination  training, 

2.  subteam  system  training, 

3.  system  subteam  training, 

4.  system  level  operational  training,  and 

5.  multiunit  system  operational  training  (Tumage,  Houser,  &  Hofinann,  1989,  p. 

A-1). 

The  five  levels  of  team  training  indicate  that  there  are  several  types  of  audiences 
for  planners  to  consider  when  developing  training.  Due  to  the  emphasis  on  aircrew, 
seaborne  and  submariner  team  training  at  sea,  it  becomes  obvious  why  the  Navy  focuses 
most  of  its  research  in  supporting  team  training.  Since  many  teams  may  not  be 
represented  fully  at  sea,  the  simulation-based  training  allows  for  more  flexibility  by 
substituting  actual  team  members  with  computer-generated  forces.  There  is  a  definite 
need  for  training  team  responsibilities  and  coordination. 

Team  cohesion  and  interaction  build  over  time.  Tumage,  Houser,  and  Hofinaim 
(1989)  view  the  mahire  team  as  one  which  can  operate  successfully  as  a  single  organism 
in  dynamic,  and  often  stressful,  environments.  In  order  to  achieve  such  a  goal  as  a 
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the  individuals  must  master  their  skills.  In  this  manner,  industry  and  academia  can  apply 
simulation-based  training  for  team  development  and  proficiency. 

Simulation-Based  Training 

This  section  examines  the  application  of  simulations,  as  well  as,  the  capabilities, 
benefits  and  limitations  of  simulations.  Often  used  in  conjunction  vwth  learning 
exercises,  simulations  assist  as  problem-solving  decision  tools  and  evaluation  devices 
(Keiser  &  Seeler,  1987).  Using  simulations,  problem-solving  and  decision-making  skills 
can  be  learned  better  in  dynamic,  complex  and  actualistic  settings  (Keiser  &  Seeler, 
1987). 

The  widespread  use  of  simulations  encompasses  a  spectrum  ranging  from 
applications  in  mining  engineering  to  virtual  medical  surgery.  A  Rand  study  classified 
simulation  applications  into  the  four  categories  represented  in  Table  1  (Shubik  &  Brewer, 
1972).  While  some  of  the  applications  may  have  changed  since  the  1972  Rand  study,  the 
categories  are  still  relevant  today. 
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TABLE  1 


SIMULATION  APPLICATIONS 


Industrial- 

Operational 

Teaching  and 
Training 

Political-Diplomatic- 

Military 

Research 

Mining 

engineering 

Urban  development 
and  planning 
strategies 

Social-economic 

analysis 

Environmental 

system 

development  and 
design 

Apparel 

processes 

Nuclear  power 
plant  training 

Strategic  wargaming 

Virtual  surgery 

Machine 

simulators 

Dispatcher  and 
operator  training  at 
electrical  utility 
companies 

Peacekeeping  and 
disaster  assistance 

Space  exploration 
equipment 

(Revised  from  Shubik  &  Brewer,  1972) 


Simulators  allow  operators  to  learn  and  practice  for  normal,  abnormal  and 
emergency  procedures  in  a  near-realistic  setting  and  provide  accurate  real-time  feedback 
(Inui,  Myojin,  &  Oba,  1993).  The  context  of  the  situation  allows  the  users  to  visualize 
the  system’s  interrelationships  much  clearer  than  abstract  concepts.  Realistic  concepts 
cue  the  user’s  abstraction  of  the  situation  so  users  can  respond  better  to  tasks  which 
enhance  their  mental  model  of  the  process  (Inui,  et  al.,  1993).  The  features  of 
simulation-based  training  which  they  identify  enables  learners  to: 

1 .  experience  “the  objective  world  in  imaginary  environments”  and  create  mental 
models  from  this. 
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2.  not  become  bored  vsdth  the  simulation,  and 


3.  grasp  concepts  easier  through  visualization  than  by  trying  to  rmderstand 
abstract  concepts  (p.  263). 

In  addition  to  the  applications  listed  in  Table  1,  Inui,  et  ah,  (1993)  also  indicate 
simulations  are  utilized  for  instruction  in  nuclear  power  plant  operations,  troubleshooting 
systems,  physics,  chemistry  and  language  education.  An  example  of  how  various  other 
domains  have  benefited  by  employing  simulations  is  provided  in  Table  2. 


TABLE  2 


SIMULATION  BENEFITS 


Event/Domain 

Result  from  Simulation  Training 

Landing  on  the  Moon  (NASA) 

Actual  training  not  possible  on  Earth 

Top  Gun  Fighter  Weapons  School 
(U.S.  Navy  in  Viemam  conflict) 

Improved  Navy  exchange  ratio  in  air 
combat  over  North  Viemam  2.4  to  12.5 

Commercial  Airlines  (U.S.) 

FAA:  Simulator  Training  alone  qualifies 
a  pilot  to  fly  a  new  airplane  for  the  first 
time  on  a  revenue  flight 

Nuclear  Power  Plants  (U.S.) 

No  significant  accidents  (after  Three  Mile 
Island)  in  military  and  civilian  operations 

Canadian  Armor  Trophy  (Military 
Armored  forces  in  Europe) 

Extensive  simulator  training  helped  U.S. 
Army  win  for  the  first  time  in  1987 

Battle  of  73  Easting  (U.S.  Army  in 
Desert  Storm) 

Participants  report  that  success  in  battle 
was  based  on  tactical  experience  gained  in 
simulations 

(Revised  from  UCF  1ST,  1993) 
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Simulations  justify  their  utility  for  training.  However,  simulations  do  not  always 
achieve  a  sigmficant  retum-on-investment  often  because  planners  have  not  matched 
capabilities  against  training  needs.  A  systematic  approach  to  simulation  training 
effectively  develops  crew  confidence  to  prepare  for  a  diverse  set  of  emergencies  and 
increases  familiarity  with  procedures  (Roth,  Mumaw,  &  Pople,  1992).  However,  the 
accomplishment  of  this  goal  does  require  the  trainer  to  review  the  simulation’s  limitations 
and  capabilities.  One  limitation  concerns  the  ability  of  simulations  to  foster  a  training 
plan  and  environment  which  will  increase  trainee  proficiency.  To  improve  human 
reliability  in  the  nuclear  power  plant  industry  and  all  simulation  application  areas,  key 
events  are  needed  to  stimulate  active  reasoning  during  simulations  (Roth,  et  al.,  1992). 

The  manager  for  the  simulation  must  ensure  that  key  events  have  realistic  meaning  to  the 
participants  and  stimulate  the  correct  responses  for  training. 

Another  limitation  of  simulations  is  that  many  degrees  of  difficulty  are  needed  as 
scenarios  change  to  support  all  training  levels  (Spurgin,  Moieni,  &,  Orvis,  1993).  The 
result  is  the  creation  and  management  of  an  exponentially-increasing  database  of 
scenarios  filled  with  exercise  details  and  objectives  (Pemberton,  Classe,  Bradley,  & 
Wilson,  1 994).  These  scenarios  are  often  site-specific,  single-domain  scenarios  which 
permit  little  flexibility  in  sharing  and  reusing  scenarios  in  multiple  domains. 

The  future  direction  of  simulations  can  best  be  illustrated  through  the  aviation 
industry  and  the  military.  Both  groups  employ  simulations  to  save  training  time  and 
reduce  risks.  Flight  simulators,  wargames  and  military  tank  simulators  are  examples.  In 
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terms  of  current  and  future  fimding,  it  appears  that  the  most  predominant  customer  for 
simulations  is  the  military.  Confronting  a  new  horizon  filled  with  multiple  global 
missions,  the  military  continues  to  invest  in  a  simulation-based  training  environment  to 
offset  drastic  cuts  in  training  areas,  units  and  persoimel.  Applying  new  technologies  in  its 
training  programs,  the  military  looks  more  to  simulations  as  an  essential  means  for 
increasing  training  readiness. 

Estvanik  (1994)  declares  that  wargame  simulations  are  used  to  test  the 
participants  on  their  strategic  and  tactical  decision-making  given  partial  information 
distorted  by  time.  Advances  in  computer  simulations  permit  the  military  “to  train  and 
equip  forces  far  more  effectively  and  efficiently  than  ever  before”  (Krepinevich,  1994,  p. 
23).  By  1997,  the  U.S.  Army  intends  to  spend  “about  $750  million  on  the  acquisition  of 
simulators  and  another  $400  million  on  research  and  development”  (U.S.  GAO,  1993,  p. 
2). 

For  similar  reasons  as  the  military,  civilian  industries  have  embarked  on  parallel 
courses  in  preparing  and  training  personnel  to  operate  in  dangerous,  stressful  and 
complicated  environments  (Keiser  &  Seeler,  1987).  As  the  level  and  complexity  of  the 
Distributed  Interactive  Simulation  (DIS)  exercises  increase  to  accommodate 
geographically-dispersed,  telecommunicated-events,  planning  needs  expand 
exponentially.  If  the  training  scenario  development  and  task  assessment  procedures  are 
integrated  with  each  other  automatically,  then  the  construction  of  DIS  simulation  training 
exercises  could  be  a  seamless,  computer-based  process  with  less  room  for  errors  and 
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more  time  for  training  (Kazarian  &  Shultz,  1994).  Tlie  transfer  of  military  simulations  to 
civilian  use  is  likely  to  follow  the  same  path  as  other  military  initiatives  similar  to 
research  programs  which  developed  the  Internet.  With  the  large  amount  of  simulation 
efforts  devoted  by  the  government,  industry  and  academia,  research  and  development 

should  be  conducted  to  ensure  effective  and  efficient  planning  of  simulation-based 
training. 

The  benefits  and  capabilities  of  simulations  have  been  demonstrated  through 
numerous  examples  in  multiple  environments.  Simulation-based  training  does  have 
limitations  which  must  be  accounted  for  in  planning  for  training  exercises.  For  the 
military  and  civilian  training  infrastructures,  future  indicators  imply  that  simulations  will 
continue  to  contribute  to  the  overall  success  of  training.  Since  the  military  is  one  of  the 

predominant  users  of  simulations  for  training,  an  example  of  a  combat  constructive 
simulation  will  now  be  discussed. 


Janus  Simulation 

For  the  purposes  of  this  study,  the  scope  of  the  training  systems  will  be  limitpH  to 
those  simulation-based  training  systems.  This  focus  is  critical  since  there  are  many 
trainers,  model  mock-ups  and  instrumentation  systems  in  the  training  systems  domain. 
One  of  the  constructive  team  training  simulations  used  extensively  by  the  U.S.  Army  and 
U.S.  Marine  Corps  is  the  Janus  simulation. 
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Janus  is  a  next-event,  stochastic,  Monte  Carlo  simulation  which  allows  exercise 
participants  to  train  on  the  command  and  control  of  their  respective  units  in  an  artificial 
combat  environment.  The  Janus  combat  simulation  program  is  written  in  FORTRAN  77 
and  fimctions  within  a  UNIX  operating  system.  The  Janus  simulation  depicts  a  military 
ground  and  air  battle  from  the  perspective  of  up  to  six  sides  offerees.  From  the  five 
levels  of  training  defined  by  Tumage,  et.  al.  (1989)  and  discussed  on  page  21,  Janus  can 
be  categorized  as  a  multi-unit  system  for  operational  training. 

The  fidelity  of  the  Janus  simulation  stimulates  players  to  visualize  battlefield 
concepts  in  dynamic,  and  seemingly  realistic  training  scenarios.  Since  the  Janus 
simulation  is  prevalent  at  many  military  installations  as  a  training  system  for  planning 
teams,  it  was  selected  for  this  study. 

This  section  has  discussed  the  merits  of  systematic  training  models,  team  trainings 
and  simulations.  Based  on  the  preceding  literature  review,  there  is  a  requirement  for  a 
training  model  designed  for  team  simulation-based  training  similar  to  training  with  the 
Janus  simulation  system.  The  next  section  will  provide  a  review  of  methods  for 
developing  traiiung  plans  which  incorporate  simulation-based  training  models. 

Training  Plan  Development 

To  examine  the  training  plan  development  area,  present  methods,  as  well  as 
building  mental  models  of  the  training  plan  development  process  will  be  detailed. 
Training  plan  development  is  the  process  of  identifying  training  goals,  objectives,  needs. 
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resources,  action  and  evaluation  for  a  particular  training  event.  Rouse  and  Howard 
(1993)  remarked  that  one  significant  problem  is  that,  in  general,  poor  plans  are  caused  by 
deficient  connections  among  objectives,  strategies,  and  agendas.  Rouse  and  Howard 
(1993)  also  contend  that  many  people  do  not  receive  training  on  how  to  create  a  well- 
developed  and  efficient  plan.  Those  who  are  not  trained  in  planning  also  have  to  research 
organizational  practices  and  procedures  in  order  to  initiate  the  process. 

Present  (Manual)  Methods 

The  need  for  a  training  planning  model  exists  to  ensure  training  objectives  are  met 
by  the  organization.  To  accomplish  the  training  for  an  organization,  Sloman  (1994) 
suggests  that  the  trainer  must  be  able  to  apply  information,  resources  and  management 
support.  As  a  result,  training  is  carefully  crafted  to  ensure  that  training  tasks  match 
training  objectives. 

Planning  is  an  information-intensive,  processing  system  which  is  very  manual. 
Instead  of  minutes  to  do  planning  for  simulation  exercises,  Pemberton,  Classe,  Bradley, 
and  Wilson  (1994)  contend  that  manual  planning  processes  require  weeks  or,  in  some 
cases,  months.  A  training  guidance  document  is  a  manually-intensive  effort  to  align  a 
training  planner’s  objectives  with  the  training  model  in  the  organization.  This  process 
assists  in  showing  the  planner  how  to  orient  training  on  the  users,  tasks  and  models. 
According  to  Tolbert  and  Bramwell  (1992),  the  training  planner’s  needs  are  satisfied  by 
such  a  tool: 
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A  training  guidance  document,  like  a  review  plan,  can  be  a  valuable  aid  in 
promoting  effective  training  practices.  It  can  improve  understanding  of 
what  constitutes  efficacious  training  and  can  be  used  to  secure  and 
maintain  optimal  levels  of  trainee  performance.  As  a  result,  reliable  plant 
performance  can  be  supported  (p.  495). 

To  provide  information  on  training  development  processes  and  teach  the  proper 
methods  for  developing  training,  Tolbert  and  Bramwell  (1992)  also  assert  that  a  training 
guidance  docmnent  should  be  the  basis  for  a  training  plan: 


1 .  The  organization  of  a  training  guidance  document  relays  important 
information  to  the  user.  It  should  be  an  accurate  portrayal  of  the  relationships 
among  all  system  elements  relative  to  training. 

2.  The  type  of  information,  amount  of  information,  and  level  of  detail  to  be 
included  in  a  training  guidance  document  depend  on  the  users  and  their  tasks. 

3.  Common  and  well-understood  terms  support  the  development  of  a  widespread 
common  model  of  intentions. 

4.  A  proper  mix  of  comprehensiveness  and  detail  facilitates  the  document’s 
user’s  tasks,  (p.  494) 

As  shown  in  the  preceding  examples,  the  manual-based  practices  in  training  plan 
development  have  been  and  are  likely  to  continue  as  the  norm  for  training  developers. 

The  end  result  has  been  masses  of  documents  and  paperwork  created,  maintained  and 
often  underutilized  throughout  the  training  planning  process.  As  planning  and  its 
inherent  complexity  are  considered,  there  appears  to  be  a  need  to  develop  an  effective 
representation  of  the  training  planning  development  process.  Future  training  plan 
development  methods  promise  to  alleviate  this  manually-intensive  workload. 
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Mental  Models  and  Training  Plan  Development 


Definition  and  development  of  mental  models 

The  theory  of  mental  models  emerged  from  several  studies  which  include  problem 
solving  research  reported  by  Newell  and  Simon  (1972),  and  Anderson’s  (1981)  work  in 
cognition.  Johnson-Laird’s  (1983)  study  asserted  that  acquiring  an  expert’s  mental  model 
of  a  task  to  guide  design  empowered  individuals  to  adopt  the  characteristics  required  to 
think  critically  by  applying  the  expert’s  mental  model  as  their  own  model  for  solving 
problems  (Chapman  &  Allen,  1994).  Dayman  and  Mayer  (1984)  termed  a  mental  model 
as  a  “metaphor  consisting  of  the  components  and  the  operating  rules  of  the  machine”  (p. 

1 89).  The  research  of  machine  operator  tasks  by  Moray  (1987)  views  mental  models  as 
“quasi-independent  subsystems  into  which  the  total  system  can  be  decomposed”  (p.  619). 

Johnson-Laird  (1983)  asserts  that  “mental  models  owe  their  origin  to  the 
evolution  of  perceptual  ability  in  organisms  with  nervous  systems”  and  thus  the  primary 
source  of  mental  models  is  perception  (p.  406).  Johnson-Laird  credits  Kenneth  Craik  as 
the  first  to  consider  modeling  mental  processes  when  Craik  “proposed  that  thinking  is  the 
manipulation  of  internal  representations  of  the  world”  (p.  x).  The  procedure  for 
confirming  statements  regarding  mental  models  emanates  firom  a  perceptual  skill  or 
activity  which  relates  written  text  to  the  environment. 

Mental  models  embrace  all  aspects  of  a  system  firom  its  physical  objects  to  its 
associated  abstractions  and  idealistic  concepts  (Lindegaard,  1987).  Mental  models  are  an 
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individual’s  representation  of,  or  “are  used  to  arrange”,  information  into  a  structure  which 
allows  versatile  storage  and  retrieval  of  information  in  terms  of  features  related  to 
situations,  entities  and  environmental  conditions  (Tannenbaum,  Cannon-Bowers,  Salas, 

&  Mathieu,  1993).  The  basis  of  a  system  is  patterned  after  the  expert’s  mental  model  is 
sometimes  called  the  “cognitive  task  model”  or  “cognitive  simulation  model” 
(conversation  with  Dr.  Kent  Williams,  March  1996).  For  this  study  the  definitions  of 
components  described  by  (Norman,  1983)  will  apply.  Norman  states  that  “in 
consideration  of  mental  models  we  need  really  consider  four  different  things:  the  target 
system,  the  conceptual  model  of  that  target  system,  the  user’s  mental  model  of  the  target 
system,  and  the  scientist’s  conceptualization  of  that  mental  model”  (p.  7).  For  the  user, 
mental  models  evolve  naturally  through  interaction  with  a  target  system  based  on  the 
user’s  background  and  expertise  (Norman).  The  scientist’s  conceptualization  of  a  mental 
model  (which  this  study  seeks  to  capture  for  automated  training  plan  development)  is  “a 
model  of  a  model”  (Norman,  1983,  p.  8). 

Johnson-Laird  (1983)  summarizes  the  development  of  mental  models  by 
explaining  that: 

1 .  they  derive  fi'om  a  relatively  small  set  of  elements  and  recursive  operations  on 
those  elements;  [and] 

2.  their  representational  power  depends  on  a  further  set  of  procedures  for 
constructing  and  evaluating  them  (pp.  429-430). 
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Johnson-Laird  (1983)  has  recognized  that  mental  models  vary  in  form,  pxirpose 
and  contents.  “The  mental  model  is  constructed  on  the  basis  of  the  truth  conditions  of  the 
propositions  expressed  by  the  sentences  in  the  discourse”  (Johnson-Laird,  p.  407).  Since 
text  can  be  imaginary  or  fictitious,  its  ability  to  represent  a  single,  well-defined  model  is 
difficult  due  to  its  propensity  to  encompass  a  multitude  of  states.  Additionally,  Johnson- 
Laird  stipulates  that  mental  models  easily  embody  certain  concepts,  while  other  concepts 
are  not  so  easily  represented  with  mental  models. 

Staggers  and  Norcio  (1993)  contend  that  mental  models  are  formed  through 
metaphors  or  analogies  to  assist  in  structuring  new  domains.  Along  with  this  premise  is 
that  people  usually  begin  building  their  mental  models  using  preconceived  analogies  fi’om 
similar  or  related  systems.  Staggers  and  Norcio  (1993)  point  out  that  even  though  there 
are  researchers  who  either  support  the  analogy  or  metaphor  approach  for  building  mental 
models,  there  are  more  important  processes  to  consider: 


Whether  one  calls  entities  analogies  or  metaphors  seems  less  important 
than  the  overall  notion  that  these  entities  serve  as  model  construction 
devices.  Instead  of  being  strict  entities  by  themselves,  these  smaller 
objects  may  serve  as  the  genesis  for  a  larger  mental  model  of  a  new 
domain  of  knowledge.  Users  may  transfer  knowledge  about  familiar 
systems  to  new  systems  in  the  form  of  visual  phenomena  known  as  either 
analogies  or  metaphors.  At  times  dissimilarities  may  be  the  impetus  for 
the  construction  of  new  relations  or  structures;  however,  users  may  not 
reject  analogies  or  metaphors  even  given  the  presence  of 
dissimilarities...(p.  591). 

The  development  of  a  person’s  mental  model  is  an  abstract  construction  of 
modeling  a  domain  of  knowledge  about  a  certain  process.  The  relationships  among  the 
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input  and  output  results  from  certain  activities  within  a  process.  The  development  of 
mental  models  reflects  a  person’s  thinking  about  situations,  but  there  are  still  aggregation 
of  models  or  processes  within  a  person’s  mental  model  which  are  unknown  and  thus 
impossible  to  capture.  The  differences  between  novices  and  experts  may  help  to  explain 
some  of  these  ideas  on  development. 

The  complexity  and  completeness  of  a  mental  model  vary  according  to  a  person’s 
background,  knowledge  and  experience.  Staggers  and  Norcio  (1993)  expect  “that  naive 
users  not  only  commit  mistakes  more  often  but  also  commit  more  serious  mistakes  than 
experts”  (p.  595).  The  expert’s  performance  overshadows  that  of  the  novice  in  that 
experts  tend  to  structure  and  recall  large  amounts  of  information  easier  than  novices 
(Schvaneveldt,  Durso,  Goldsmith,  Breen,  &  Cooke,  1985). 

In  computer  programming  Canas,  Bajo,  and  Gonzalvo  (1994)  report  that  “the 
conceptual  structures  of  novices  and  expert  programmers  have  indicated  that  experts  not 
only  possess  more  programming  knowledge  than  novices,  but  [experts]  also  organize  the 
knowledge  differently”  (p.  796).  The  experts  tend  to  synthesize  their  programming 
knowledge  for  a  deeper  understanding  and  application,  while  novices  show  superficial 
knowledge  of  the  programming  language.  Staggers  and  Norcio  (1993)  support  this 
premise  by  stating  that  many  authors  concur  that  the  expert’s  mental  models  are  abstract, 
while  the  novices’  models  are  more  concrete.  Experts  possess  the  ability  to  merge 
imaginary  entities  with  real  entities  in  order  to  create  and  refine  higher  level  conceptual 
models  (Staggers  &  Norcio). 
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Research  in  computer-aided  mental  modeling  indicates  that  the  designer’s 
conceptual  models  revealed  through  user  interface  design  would  enable  novices  to 
“develop  better  mental  models  and  thus  perform  tasks  more  like  experts”  (Shih  &  Alessi, 
1994).  A  novice  could  apply  the  mental  model  of  the  expert  to  begin  responding  or 
acting  as  an  expert.  The  designer  must  not  only  design  a  system  which  increases  the 
“users’  abilities  to  create  appropriate  mental  models”  but  also  know  how  the  users  think 
about  the  system  (Staggers  &  Norcio,  1993,  p.  595). 

The  breakdowns  in  the  development  of  mental  models  must  also  be  considered 
when  examining  how  mental  models  are  formed.  In  fostering  mental  models,  Staggers 
and  Norcio  (1993)  emphasize  that  designers  build  a  model  which  increases  users’  abilities 
to  create  a  mental  model  of  the  system.  Johnson-Laird  (1983)  aptly  realized  that 
developing  a  usable  and  complete  mental  model  theory  would  include  major  constraints 
which  result  from: 

1 .  the  perceived  and  conceived  stracture  of  the  world, 

2.  conceptual  relations  governing  ontology,  and 

3.  the  need  to  maintain  a  system  free  from  contradictions  (p.  430). 

Mayer  (1989)  defines  conceptual  models  as  those  which  create  mental  models 
through  text  and  graphics  that  depict  the  major  activities,  causalities  and  components  in  a 
system.  Mayer  asserts  that  “conceptual  models  can  provide  an  assimilative  context  for 
students  to  build  useful  mental  models”  (p.  61).  If  the  conceptual  models  are  ill-defined 
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or  incomplete,  the  resultant  mental  model  in  the  subject  or  student  is  also  inaccurate.  A 
mental  model  may  not  always  match  the  complete  and  accurate  mental  model  that  the 
designer  intended.  Canas,  Bajo,  and  Gonzalvo  (1994)  suggest  that  “a  mental  model 
might  contain  more  or  less  details  of  the  components  and  rules  of  the  system  and  may 
match  more  or  less  closely  the  real  characteristics  of  the  system”  (p.  795).  The  user  may 
have  preconceived  ideas  and  mental  models  which  supplement  or  fill  in  missing 
components  of  the  modeled  system’s  process. 

The  development  or  evolution  of  mental  models  in  people  adhere  to  these 
constraints.  Everyone  understands  and  perceives  conceptual  and  physical  models 
concepts  in  a  different  manner  than  others  may.  A  person’s  mental  model  is  the 
understanding  of  the  relationships  between  the  input  and  the  output  to  predict,  after  the 
development  of  an  accurate  mental  model,  the  output  which  could  be  produced  for  the 
possible  inputs  (Eberts,  1994). 

Johnson-Laird  (1983)  proposed  that  the  theoretical  aspect  of  mental  models 
should  result  in  a  complete  and  accurate  model.  However,  the  mental  model’s  content  is 
“both  a  function  of  the  model  and  the  processes  that  evalxiate  it”  (Johnson-Laird,  1983,  p. 
408).  The  iterative  construction  of  mental  models  results  from  reasonable,  yet  biased, 
assimptions.  The  model  is  recursively  modified  if  the  assinnptions  are  incorrect.  The 
end  result  is  that  the  model  may  embed  inconsistencies  which  do  not  readily  reveal 
themselves. 
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Another  aspect  of  recursive  iteration  is  vv^hat  happens  when  users  stop  forming 
their  mental  models  of  the  process.  Moray  (1987)  hypothesizes  that  skilled  operators 
develop  a  rigid  and  inflexible  mental  model  once  they  are  content  with  their  level  of 
learning.  The  danger  here  is  that  the  unbending  mental  model  shatters  when  an  unknown 
situation  is  encountered.  “Instead  of  being  triggered  by  discrepancies  to  pursue 
induction,  they  [operators]  desperately  try  to  fit  data  to  their  existing  models  and  are 
typically  very  inflexible”  (Moray,  p.  628).  In  this  case,  the  mental  model  ceases  to 
develop  into  a  complete  and  mature  representation  of  the  entire  environment  in  which  the 
operator  must  function. 

There  are  several  ways  already  known  to  enhance  the  completeness  and  accuracy 
of  mental  models.  First  of  all,  the  conceptual  model  of  the  target  system,  such  as  the 
training  planning  development  process,  should  be  designed  using  subject  matter  experts 
(SME)  (Norman,  1983).  If  there  is  a  lack  of  user  or  SME  input  in  the  design  process,  the 
system  will  only  partially  convey  the  intended  model  and  most  likely  confuse  the  users. 

Secondly,  the  mental  model  should  be  verified  to  determine  that  it  follows  and 
includes  the  components  of  the  process  imder  consideration.  To  verify  the  mental  model, 
a  system  is  needed  for  recording  the  tasks,  input  and  output  variables,  and  the 
relationships  in  the  process.  In  this  manner,  the  verification  and  implementation  of  the 
system  will  evaluate  the  flow  of  actions  triggered  by  a  correct  mental  model. 

There  are  several  methods  available  for  conveying  the  conceptual  model  of  a 
system  for  users/evaluators  to  create  their  own  mental  models.  The  Goals,  Operators, 
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Methods  and  Selection  (GOMS)  methodology  is  a  hierarchical  goal  decomposition 
method  proposed  by  Card,  Moran,  and  Newell  (1983).  “The  GOMS  analysis  technique  is 
also  compatible  with  the  recall  of  procedural  knowledge  since  it  is  based  upon  the 
production  theory  of  learning  and  memory  which  has  received  empirical  support  in  the 
research  literature”  (Williams  &  Kotnour,  1992,  p.  57).  GOMS  is  one  of  the  procedures 
which  enables  the  identification  of  conceptual  models  of  target  system  tasks  using 
knowledge  statements  which  describe  a  process. 

Cognitive  complexity  models  describe  the  essential  procedural  knowledge  in 
terms  of  a  production  system  architecture.  Bovair,  Kieras,  and  Poulson  (1990)  state  that 
“like  the  GOMS  model,  cognitive  complexity  models  are  descriptions  of  the  knowledge 
required  to  use  systems  and  are  not  intended  to  be  complete  simulations  of  the  actual 
mental  processes  of  the  user”  (p.  5).  Bovair,  et  al.  applied  the  production  system  as  the 
basis  of  the  cogmtive  complexity  model  due  to  its  widespread  acceptance  and  success  by 
other  researchers. 

The  production  system  includes  “a  working  memory,  a  collection  of  production 
rules,  and  an  interpreter”  (Bovair,  et  al.).  Each  of  the  components  of  the  production 
system  function  to  model  the  way  humans  think.  The  working  memory  represents 
current  environmental  goals  and  inputs,  as  well  as,  previous  information  which  explains 
previous  and  current  states  of  activity.  The  production  rule  is  an  IF-THEN  statement 
indicating  condition-action  pairs  to  describe  the  external  environment  or  working 
memory.  Switching  between  recognition  and  action  cycles,  the  interpreter  evaluates  rules 
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and  acts  in  a  predetermined  maimer.  The  production  rule  system  has  been  used  in  many 
systems  in  applied  cognitive  research  (Anderson,  1987). 

How  mental  model  theory  has  been  applied  to  design  in  the  past 

Mental  models  evolve  from  the  user’s  previous  experience,  knowledge  and 
perceptions  related  to  a  system  (Wilson  &  Rutherford,  1989).  Evaluation  of  users’ 
mental  models  is  frequently  accomplished  by  measuring  learning,  efficiency  workload  or 
time  performance.  Qualitative  and  subjective  evaluation  methods  query  whether  the 
subjects  were  able  to  increase  their  understanding  of  the  particular  process. 

Dayman  and  Mayer  (1984)  declared  success  in  conveying  the  conceptual  model  of 
a  calculator  stating  that  “mental  models  can  be  explicitly  taught  to  users  of  electronic 
computing  devices,  and  that  these  models  can  enhance  the  level  of  sophistication  of  user 
performance”  (p.  198).  The  initial  results  showed  success,  but  there  was  still  a  need  to 
improve  the  calculator  conceptual  model  for  a  broader  base  of  users.  Dayman  and  Mayer 
suggest  that  further  research  is  required  to  iterate  and  find  the  best  conceptual  model  for 
helping  more  people  imderstand  how  calculators  operate. 

A  mental  model  proposed  by  Chapman  and  Allen  ( 1 994)  exercised  cognitive  task 
analysis  research  for  designing  mass-production  of  maintenance  multimedia  courseware 
simulations.  The  authors  declared  success  when  they  were  able  to  demonstrate  the 
improvement  of  users’  mental  models  through  teaching  cognitive  troubleshooting 
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strategies  in  actual  situations.  Verification  of  the  mental  model  was  achieved  by  applying 
the  tasks  in  a  realistic  environment. 

McAlindon  (1994)  reported  successful  results  in  using  mental  model  theory  to 
represent  the  linear  QWERTY  keyboard  structure  to  build  his  patented  Keybowl.  The 
mental  model  of  the  QWERTY  keyboard  was  identified  and  then  used  in  modeling  the 
Keybowl  to  determine  where  to  place  the  characters  and  convey  the  mental  model  of 
typists.  The  Keybowl  typists  indicated  that  Keybowl  typing  was  easy  to  learn  because 
of  its  QWERTY  compatible  mental  model  coded  character  schema  and  visible  character 
arrangement”  (McAlindon,  p.  166). 

For  the  definition  of  success  in  a  mental  model  theory  application,  there  are  many 
possibilities  relevant  to  all  domains.  However,  sometimes  the  results  are  convoluted  with 
external  or  internal  factors  which  fail  to  reveal  the  importance  of  the  mental  model 
theory. 

One  of  the  most  intriguing  factors  involved  in  determining  a  user’s  mental  model 
is  the  transfer  of  knowledge.  A  mental  model  may  not  change  significantly  until  major 
increases  in  learning  have  occurred.  This  interesting  feature  arose  from  the  study  by 
Canas,  Bajo  and  Gonzalvo  (1994)  in  that  the  novice  programmers  who  received  limited 
programming  instruction  in  the  test  did  not  improve  performance.  More  programming 
instmction  and  learning  were  deemed  essential  to  enable  the  novice  programmers  to 
adequately  improve  their  mental  models  of  the  programming  process. 
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Relating  the  concept  of  mental  models  to  team  planning.  Stout  (1994)  sought  to 
show  how  shared  mental  models  enabled  team  members  to  understand  one  another’s 
informational  requirements.  The  difficulty  here  was  capturing  not  only  one  mental 
model,  but  understanding  and  finding  the  synergism  of  several  mental  models  for  team 
members  who  fimction  in  distinct,  yet  contributive  tasks.  Team  training  planning  is  an 
area  in  which  the  literature  is  not  extensive,  and  there  are  only  a  few  “empirical 
investigations  of  planning  in  teams”  (Stout,  p.  63).  Stout’s  team  research  is  a  good  start, 
but  more  work  needs  to  be  done  in  multiple  mental  models  before  success  can  be 
declared  in  this  area. 

How  mental  model  theory  can  be  applied  to  the  planning  process 

Developing  accurate  mental  models  are  critical  to  planning  for  simulation-based 
training.  The  computer-based  tool  incorporates  and  transfers  the  appropriate  conceptual 
model  to  the  novice  planner.  To  enable  a  novice  trainer  to  embrace  some  of  the  training 
plan  developer’s  expert’s  knowledge,  it  is  essential  to  capture  the  expert’s  mental  image 
of  training  plan  development.  Computer-based  training  planning  tools  are  now  examined 
to  determine  their  utility  and  feasibility  for  assisting  novices  in  forming  a  mental  model 
of  training  plan  development. 

The  complexities  of  concepts  and  actions  within  the  systematic  training  planning 
process  for  simulation-based  training  warrant  the  application  of  an  expert’s  mental 
models  in  the  design  of  computer-based  planning  tools.  Rouse  and  Morris  (1986)  assert 
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that  complex  situations  such  as  developing  training  strategies  are  ideal  for  mental  model 
theory  application.  Pietras  and  Coury  (1994)  affirm  that  a  cognitive  model  of  human 
planning  behavior  could  represent  the  planning  process. 

“Managerial  planning  tasks,  usually  referred  to  as  strategic  or  corporate  planning, 
involve  a  series  of  mental  processes  preceding  or  leading  to  the  taking  of  important 
decisions  about  the  future  course  of  a  firm”  (Marmaras,  Lioukas,  &  Laios,  1992,  p.  1222). 
The  criticality  of  planning  for  simulation-based  training  has  equal  importance  for 
training  to  perform  in  stressful  and  dangerous  situations.  The  transformation  of  training 
planning  concepts  suggests  an  extension  of  mental  models  to  build  the  expert’s  view  of 
the  system.  In  a  training  planning  system,  the  expert’s  view  or  mental  model  generally 
follows  the  heuristic  systematic  training  planning  steps  to  ensure  appropriate 
relationships  and  procedures  are  accomplished. 

For  training  plan  development,  a  variety  of  tasks  and  objectives  are  performed  by 
the  planner  in  organizing,  executing  and  refining  effective  training.  The  development  of  a 
mental  model  which  follows  an  accurate  model  of  the  training  plan  development  process 
seeks  to  transfer  the  knowledge  through  an  automated  process  versus  conventional 
manual  methods.  “Computer  users  are  gener^ly  required  to  acquire  the  structural 
information  of  systems  and  individual  tasks  in  a  manner  which  is  incidental  to  their  main 
task  objectives”  (Stanney  &  Salvendy,  1994,  p.  607).  The  system  requires  the  user  to 
provide  certain  information  in  order  to  conduct  the  tasks  which  fulfill  the  major  requisites 
of  the  system.  Kline,  Blickensderfer,  Dryer  and  Bochenek  (1995)  reiterate  this  point  by 
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stating  that  “as  the  system  queries  the  user  for  information  about  the  problem,  the  user 
should  begin  to  build  a  mental  model  of  what  the  system  is  doing”  (p.  195). 

The  proper  design  of  an  automated  planning  tool  enables  the  novice  to  build  a 
mental  model  of  the  training  planning  process.  The  validation  of  this  statement  requires 
examination  of  how  automated  planning  tools  are  better  than  manual  planning  methods 
for  training  planning,  as  well  as,  the  effects  that  an  automated  planning  tool  have  on  the 
mental  model  of  a  novice  planner. 

Corroborated  by  Merrill,  Li  and  Jones  (1989),  the  need  for  computer-based 
programs  is  permeating  all  domains  of  industry: 


The  impact  of  computerization  on  other  fields  has  been  to  increase 
productivity  by  reducing  labor  costs,  or  allowing  greater  production  from 
the  same  labor.  Personal  computers  probably  owe  their  success  to  the 
electronic  spreadsheet.  Every  financial  planner  could  immediately  see  the 
efficiency  of  using  an  electronic  spreadsheet.  Tasks  that  at  one  time  might 
require  days  or  weeks  could  now  be  accomplished  in  minutes  or  hours  (p. 

10). 

Similar  to  the  preceding  statement,  there  are  currently  many  promises  whereby 
technology  is  seen  as  a  viable  means  for  increasing  the  efficiency  of  conducting  routine 
tasks.  For  planning,  Tolbert  and  Bramwell  (1992)  remark  that  computer-based  planning 
tools  are  essential  to  guide  the  plaiming  process.  Tolbert  and  Bramwell  outline  an 
approach  which  defines  training  guidance  documents  as  a  methodology  for  organizing  the 
the  training  planning  process.  However,  the  proposed  process  is  too  complex  and 
intensive  to  rely  on  manual  procedures  and  techniques.  When  training  and  planning  are 
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coupled  with  simulations,  the  manual  requirements  become  an  unmanageable  burden  of 
input  and  output  variables  intertwined  into  relationships  within  the  modeled  process. 
Table  3  highlights  the  functions  in  automated  planning  tools  for  simulation-based 
training. 


TABLES 

AUTOMATION  ROLES  IN  TRAINING  PLAN  DEVELOPMENT 


Area 

Function 

Procedure 

Remembering  the  essential  steps  in  training  plan  development 

Planning  context 

Visualizing  (or  vmderstanding)  a  training  plan  as  it  unfolds 

Information 

management 

Filing,  storing  and  retrieving  training  plans  and  scenarios  using 
routine  and  systems  design  principles 

Automation 

Using  technology  to  solve  a  problem 

Simulation-based 

training 

Automating  the  efforts  normally  reqviired  for  rebuilding  and 
initializing  simulation  scenarios 

Editing 

Incorporating  changes  to  training  plans 

Training  status 

Updating  proficiency  ratings 

The  abimdance  of  information  and  processing  activities  in  the  training  plan 
development  process  warrants  automated  assistance  to  ease  the  cognitive  or  mental 
burden  on  the  user.  Helander  (1988)  states  that  “cognitive  psychologists  have  viewed 
human  cognition  as  an  information-processing  activity  with  many  similarities  to 
computer  systems”  (p.  893). 
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Given  that  it  would  require  a  novice  a  considerable  amount  of  time  and  numerous 
opportumties  of  exposure  to  the  training  plan  development  process  for  simulation-based 
training,  the  use  of  a  computer-based  system  for  planning  training  is  viable.  The 
cognitive  and  perceptual  requirements  for  the  training  planning  process  require  the 
training  planner  to  utilize  the  functions  listed  in  Table  3.  In  order  to  visualize  and  focus 
on  this  process,  automated  planning  tools  will  enhance  the  novice’s  abilities  to  adopt  the 
expert’s  view  of  the  training  plan  development  process. 

Since  training  plan  development  encompasses  the  arrangement  of  information  for 
a  specific  environment,  the  use  of  mental  models  theory  in  designing  planning 
information  for  training  merits  further  research.  The  associated  input  and  output  and 
relationships  of  goals  and  plans  are  the  key  elements  of  the  training  plan  development 
process.  Using  a  cognitive  analysis  approach,  production  rules  can  be  developed  fi-om 
these  elements  to  represent  the  mental  model  of  an  expert  training  plan  developer  of 
simulation-based  training. 

Helander  (1988)  alleges  that  a  computer-based  mformation  system  should  support 
the  development  of  the  mental  models  essential  for  conducting  the  functions  of  a  process. 
The  requirement  to  acquire  a  mental  model  of  training  planning  is  critical  for  success  in 
simulation-based  traimng.  The  planning  tool’s  display  structure  must  accommodate 
fostering  and  building  of  mental  models  for  novices  and  inexperienced  users  to 
understand  the  training  planning  process. 
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In  order  to  assist  users  in  building  an  appropriate  mental  model,  the  interface 
design  must  convey  information  on  the  relationships  and  processes  within  the  training 
plan  development  process.  Building  a  mental  model  of  an  interface  can  reduce 
complexities  in  an  interface  if  the  number  of  operations  to  accomplish  a  task  are 
consolidated.  For  computer  interfaces,  Eberts  (1994)  espouses  that  three  areas  are 
effective  for  developing  concise  mental  models  of  a  system: 

1 .  active  interaction  with  the  computer, 

2.  the  use  of  metaphors  and  analogies  to  explain  the  system,  and 

3.  the  use  of  spatial  information  to  convey  the  conceptual  model  (p.  162). 

First  of  all,  in  order  to  develop  a  concise  mental  model  of  the  training  planning 

system,  users  need  to  be  able  to  explore  and  test  the  effects  of  input  actions  on  the 
computer  without  being  penalized  by  time  losses  or  serious  errors  (such  as  loss  of 
program  control).  Eberts  (1994)  also  suggests  that  words  and  symbols  induce  the  users  to 
explore  the  system  further.  A  training  planning  system  would  apply  recognizable, 
domain-specific  vocabulary  and  icons  to  enable  the  users  to  begin  building  a  mental 
model  of  the  designer’s  conceptual  model  of  the  system. 

The  next  consideration  for  designing  computer-based  tools  using  conceptual 
models  is  that  a  training  planning  system  presents  users  with  metaphors  and  analogies 
which  will  teach  the  conceptual  system’s  mechanics  and  operations.  Metaphors  and 
analogies  based  on  concrete  images  related  to  the  training  plan  development  process 
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enable  novices  to  accomplish  tasks  that  require  synthesizing  training  planning 
knowledge.  One  metaphor  in  a  computer-based  planning  tool  could  be  a  clipboard  to 
represent  the  process  of  assessing  task  accomplishments. 

The  third  important  system  feature  requires  spatial  representation  of  the  designer’s 
conceptual  model.  Presenting  a  “picture”  of  the  system’s  components,  inputs  and 
outputs,  or  overall  structure,  graphical  representations  seek  to  assist  the  novice  in 
visualizing  the  designer’s  conceptual  model.  This  feature  implies  that  the  automated 
plaiming  tool’s  layout  permits  the  user  to  visualize  the  training  plan  development  process 
since  the  computer-based  tool  guides  the  user  through  the  process  of  completing  the 
planning  tool’s  steps  and  procedures.  Consequently,  as  users  become  more  familiar  with 
the  procedures,  they  will  begin  to  leam  and  visualize  what  the  inputs  and  outputs  are  in 
relation  to  the  training  plan  development  process  without  having  to  follow  all  of  the  steps 
in  the  system. 

In  a  training  planning  system,  experts  must  be  able  to  convey  the  process  to 
novice  users.  Managers  and  corporate  leaders  frequently  task  novices  to  conduct  training 
planning  activities.  Hollnagel,  Hoc,  and  Cacciabue  (1995)  state  that  “expertise  can  be 
defined  as  the  knowledge  that  a  person  can  bring  to  bear  on  a  situation”  (p.  279).  A  user 
interface  design  which  mimics  the  expert’s  conceptual  structure  of  a  training  planning 
process  thus  may  enable  the  novice  to  apply  the  same  knowledge  an  expert  would  use  for 
training  plan  development. 
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In  a  manually-oriented  method,  the  novice  training  planner  follows  written 
guidance  documents  and  procedures,  but  fails  to  understand  or  grasp  the  essence  of  the 
plaiming  exercise  until  gaining  sufficient  experience  in  conducting  planning  for 
simulation-based  training.  The  computer-based  training  planning  tool  may  enable  the 
novice  to  learn  quickly  the  training  development  steps  while  visualizing  the  process  and 
receiving  cues  to  guide  progress.  The  implication  is  that  if  a  training  planning  tool 
closely  mirrors  the  expert’s  view  of  training  planning  development,  then  the  novice 
planner  may  create  an  accurate  mental  model  (Eberts,  1994).  Therefore,  the  automated 
planning  tool  should  attempt  to  model  the  training  plan  development  process  in  such  a 
manner  that  the  tool’s  interface  conveys  the  expert’s  mental  model  of  how  training  plan 
development  works.  As  discussed  in  the  next  section,  there  are  important  shortcomings 
and  weaknesses  to  consider  in  this  area. 

Mental  model  theory  weaknesses/potential  pitfalls 

The  most  significant  weaknesses  and  potential  pitfalls  in  using  mental  model 
theory  emcompass  the  user’s  mental  model  development.  Johnson-Laird  (1983) 
stipulates  that  the  concepts  of  “time,  space,  possibility,  permissibility,  causation  and 
intention”  (p.  413)  affect  how  users  would  comprehend  a  conceptual  model  of  a  system. 
The  textual  and  graphical  references  in  the  software  planning  tool  may  be  insufficient  to 
create  an  accurate  and  complete  mental  model.  Consequently,  an  individual’s  mental 
model  may  function  to  a  certain  level,  but  not  as  complete  as  intended  by  the  designer. 
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The  user’s  mental  model  of  the  system  should  be  verified  and  checked  prior  to 
assuming  that  the  novice  or  inexperienced  user  fully  grasps  the  conceptual  representation 
of  the  designer.  The  need  for  verification  of  the  user’s  mental  model  applies  to  the 
training  plan  development  process.  The  backgroimd,  knowledge  and  experience  of  the 
training  planners  affect  their  perception  of  the  mental  model’s  semantics  and 
representations  of  the  training  planning  system.  The  results  may  be  inconsistency  and 
inaccuracy  in  development  of  a  training  plan. 

Next,  the  designer  is  also  responsible  for  the  development  of  conceptual  models. 
The  construction  of  a  universally-acceptable  conceptual  model  is  firequently  unattainable. 
Eberts  (1994)  states  that  one  problem  with  GOMS  is  that  “different  task  analyzers  may 
develop  different  task  analyses  for  the  same  task  (p.  365).  There  is  a  need  to  build  on 
several  iterative  revisions  of  the  conceptual  model  to  develop  an  approximation  of  the 
process.  It  is  the  designer’s  responsibility  to  consolidate  these  revisions  and  create 
systems  which  enable  users  “to  develop  more  coherent,  useable  mental  models” 

(Norman,  1983,  p.  14).  For  training  plan  development,  there  are  many  experts  and  task 
analyzers  who  have  different  interpretations  of  the  training  plan  development  process. 
However,  these  differences  need  to  be  reduced  by  compromise  and  adherence  to  the 
fundamental  concept  of  the  systematic  training  model  with  planning  enhancements. 

Last  of  all,  the  experimenter’s  ability  to  measure  the  initial  and  resultant  mental 
models  of  users  may  lead  to  inconsistencies  in  recursively  designing  the  conceptual 
model  portrayal  to  the  user.  There  are  methods  to  determine  whether  the  user’s  mental 
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model  is  developing  as  the  designer  had  intended,  but  they  sometimes  do  not  reveal 
practical  relevance  to  a  dynamic  or  evolving  system.  Inaccurate  assessments  could  affect 
the  mental  model’s  production  and  understanding.  Consequently,  the  user,  designer  and 
experimenter  must  work  together  to  develop  and  evaluate  the  system  which  will  convey 
the  most  accurate  and  complete  conceptual  model. 

Computer-Based  Training  Planning  Tools 

Currently,  training  proficiency,  task  selection  and  prioritization,  training 
conditions  and  effectiveness  measures,  must  also  be  processed  and  integrated  by  manual 
planning  methods  to  match  training  needs  against  training  simulation  capabilities, 
limitations,  and  resource  availability.  This  time-consuming,  complex  process  often 
discourages  the  novice  training  plaimers  from  considering  the  use  of  simulation-based 
training. 

There  are  both  planning  tools  and  training  planning  tools  which  use  computers. 
The  planning  tools  in  today’s  market  vary  from  generalized  project  planning  packages  to 
specialized  decision  support  programs. 

General  Planning  Tools 

The  category  of  general  planning  tools  does  not  focus  on  any  one  component  of 
planning  or  its  relation  to  domains  such  as  training.  Instead,  the  overall  review  of  general 
planriing  tools  is  explored  to  determine  how  planning  tools  can  assist  the  user  in 
conducting  tasks  in  the  organization.  “Tools  for  decision  support  and  scenario  analysis 
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allow  for  a  uniform  merger  between  spread-sheets,  databases,  financial  modeling 
systems,  etc.”  (Parsaye,  Chignell,  Khoshafian,  &  Wong,  1989,  p.  421).  Similar  to 
planning  systems,  decision  support  systems  are  considered  “a  special  kind  of  information 
management,  retrieval,  and  utilization,  where  the  information  is  tailored  towards 
particular  decision  making  activities”  (Parsaye,  et  al.,  1989,  p.  421). 

For  “corporate  and  strategic  planning,  tasks  involve  a  series  of  mental  processes 
preceding  or  leading  to  the  making  of  important  decisions  about  the  future  course  of  a 
firm”  (Marmaras,  Lioukas,  &  Laios,  1992,  p.  1223).  The  overall  objectives  of  a  planning 
tool  are  to  expand  on  these  features  incorporating  information  manipulation  for  the 
purpose  of  meeting  goals  and  objectives. 

Pietras  and  Coury  (1994)  assert  that  no  one  has  investigated  the  “potential  utility 
of  theoretical  research  in  planning”  (p.  6),  e.g.,  for  developing  project  management  tools. 
With  similar  considerations  in  the  training  model  and  its  planning  process,  these  tools 
could  also  be  used  for  training  plan  development.  Pietras  and  Coury  (1994)  contend  that 
their  cognitive  model: 

1 .  identifies  the  goal  and  plan  structure  of  the  planning  process,  and 

2.  captures  the  levels  of  reasoning  and  abstraction  inherent  in  planning  for 
project  management,  (p.  6) 

Project  planning  packages  contain  information  to  track  performance,  costs  and 
schedules  of  programs  (S?nn,  1989).  Some  of  the  characteristics  for  successful  project 
planning  relate  to  the  discussions  of  training  planning.  There  are  various  types  of 


50 


planning  methods  in  the  sciences.  “Mathematical  programming  has  been  traditionally 
used  for  planning  large  and  complex  systems.  Some  examples  include  air-traffic  control 
management,  scheduling  power  generation  and  transmission,  transportation  planning,  and 
manufacturing  and  distribution  scheduling”  (Zenios,  1995,  p.  195). 

Management  of  information  systems  often  involves  various  planning  techniques. 
Developed  in  1 958  for  the  U.S.  Navy  by  Booz,  Allen  and  Hamilton,  the  Program 
Evaluation  and  Review  Technique  (PERT)  is  one  of  the  most  practical  planning  methods 
(Senn,  1989).  The  PERT  program  is  incorporated  into  software  packages  such  as 
Microsoft  Project©  and  CA-SuperProject©. 

For  an  organization  or  corporation  to  build  a  plan,  the  process  continues  to  be  an 
elusive  and  fimstrating  one.  Rouse  and  Howard  (1993)  contend  that  computer-based 
planning  tools  can  significantly  improve  the  effectiveness  and  efficiency  of  plan 
generation.  To  illustrate  this  point.  Rouse  and  Howard  (1993)  have  built  a  computer- 
based  planning  tool  to  assist  corporations  with  the  complexities  associated  with  corporate 
marketing  strategic  planning.  The  primary  focus  for  this  application  is  on  plaiming 
methodologies  for  marketing  strategies  and  is  “augmented  with  on-line  tutoring  and 
expert  advice”  (Rouse  &  Howard,  1993,  p.  52). 

Some  research  references  there  is  potential  for  applying  computer-based  tools  to 
alleviate  the  burdens  of  information  overload.  Kelleher  (1993)  exploits  the  potential  for 
applying  computer-based  plaiming  tools  for  information  processing  systems. 
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Planning  methods  and  procedures  abound  in  industry,  academia  and  government, 
but  there  are  few  planning  tools  designed  to  prepare  users  and  managers  for  the 
simulation-based  training.  For  many  planners  and  users  throughout  industry,  training 
simulation  scenario  preparation  requires  hundreds  of  hours  for  preparation,  execution  and 
review.  The  goals,  plans  and  levels  of  abstraction  in  training  planning  are  similar  to  those 
in  project  management,  suggesting  the  potential  for  use  of  this  concept  for  future  training 
plan  development  methods.  The  project  management  planning  techniques  reflect  those 
used  in  training  plan  development. 

Existing  Training  Planning  Tools 

This  section  will  highlight  research  and  developments,  in  both  the  civilian  and 
military  sectors,  regarding  the  use  of  planning  tools  for  simulation-based  training. 
Throughout  industry,  analysts  and  observers  assert  that  technology  and  automation 
should  enable  people  to  conduct  routine  tasks  (Hollnagel,  Hoc,  &  Cacciabue,  1995).  For 
these  reasons,  the  implementation  of  computer-based  planning  is  essential  to  simulation- 
based  training.  Relative  to  planning,  Tolbert  and  Bramwell  (1992)  remark  that  computer- 
based  planning  tools  are  essential  to  guide  the  planning  process.  Researchers  have  only 
started  to  address  the  importance  of  using  computer-based  planning  tools  to  prepare  for 
simulation-based  training.  This  situation  is  reflected  in  the  civilian  sector.  On  the  other 
hand,  the  military  is  beginning  to  prototype  several  computer-based  approaches  to  assist 
in  the  planning  of  simulation-based  training. 
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Civilian 


There  are  already  instances  of  automated  training  planning  tools  being  developed 
in  academic  instruction  and  training.  Drewes  and  Gonzalez  (1994)  discuss  a 
methodology  for  assisting  instructors  in  monitoring  students  training  on  flight  simulators. 
This  methodology  includes  real-time  evaluation  feedback  to  allow  instructors  to  monitor 
the  training  with  the  use  of  template-based  training  goals,  subgoals  and  standards.  The 
components  of  this  computer-based  training  system  encompass  training  performance 
monitoring,  on-line  comparisons  against  standards,  and  immediate  feedback  to  the 
instructor  or  student.  All  three  of  these  functions  are  automated  as  part  of  the  training 
plan  development  process. 

Data  analysis  and  measurement  of  meaningful  results  also  require  improvement. 
The  nuclear  power  plant  industry  is  exploring  ways  to  improve  this  area  in  the  use  of 
simulations  for  training  on  power  plant  accidents  (Spurgin,  Moieni,  &  Orvis,  1993;  Roth, 
Mumaw,  &  Pople,  1993).  The  training  goals  seek  to  build  realistic  and  stressful  exercises 
for  nuclear  power  plant  incidents.  Roth,  et  al.,  (1993)  have  developed  a  systematic 
training  approach  for  scenario  development  for  simulator  training.  Their  objective  is  to 
maximize  the  current  use  of  simulator  training  time  by  infusing  mentally  challenging 
training  scenarios  into  the  operators’  training  program.  Spurgin,  et  al.  (1993)  have  sought 
to  integrate  simulator  data  collection  methods  and  training  effectiveness  measures  in 
order  to  assess  crew  performance  in  simulator  exercises.  These  are  just  some  of  the  ideas 
for  automated  training  planning  tools  in  training  evaluation  and  review  systems. 
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There  were  only  a  few  examples  of  research  and  work  in  computer-based 
planning  for  the  civilian  industry.  For  the  military,  initial  efforts  are  surfacing,  but  the 
verdict  as  to  the  success  or  contribution  of  computer-based  training  planning  tools  is  still 
speculative. 

Military 

In  recent  years,  the  Navy’s  Automated  exercise  Preparation,  Evaluation  and 
Preview  (APEP)  training  software  program  has  led  the  way  in  automated  systems  for 
training  planning  (Pemberton,  Classe,  Bradley,  &  Wilson,  1994).  APEP  was  originally 
proposed  by  Pemberton,  Campbell,  and  Ahlers  (1992)  as  a  system  which  would  generate 
scenarios  by  identifying  and  utilizing  a  “large  amount  of  information  to  simulate  a 
military  engagement”  (p.  2). 

Paralleling  the  APEP  efforts  is  a  prototype  planning  program  being  developed  by 
the  U.S.  Army.  The  Army’s  Training  Exercise  Development  System  (TREDS)  described 
in  Crissey,  Stone,  Briggs,  and  Mollaghasemi  (1994)  is  an  initiative  to  precipitate  training 
plan  development  for  simulations  by  capitalizing  on  computer  technologies.  The 
theoretical  perspective  on  which  both  APEP  and  TREDS  emanated  resemble  a  cognitive 
model  of  instructional  system  design  as  described  by  Gropper  and  Ross  (1987). 

The  Automatic  Scenario  Generator  (ASG)  is  a  predecessor  to  APEP.  Specifically, 
this  theory  of  scenario  development  proposed  the  following  structure  for  training  system 
scenario  generation  and  modification: 
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1 .  establishing  training  objectives, 


2.  determining  performance  measurement  criteria, 

3.  identifying  initial  conditions,  and 

4.  preparing  a  timeline  of  events  or  track  scripting  (Pemberton,  et  al.,  1992,  p.  3). 

The  ASG  program  initiated  by  Pemberton,  et  al.,  (1992)  was  later  converted  to 

the  APEP  planning  tool  with  limited  features  and  functionalities  in  scenario  generation. 
Pemberton,  et  al.,  (1992)  also  outlined  in  the  ASG  proposal  the  requirements  for  subject 
matter  experts  (SME)  or  instructors  to  conduct  tasks  analyses,  develop  creative  and 
imaginative  scenarios  for  students,  and  measure  student  performance.  The  objective  was 
that  automation  would  reduce  the  planning  time  and  efforts  of  training  supervisors, 
subject  matter  experts  (SME)  and  other  training  personnel.  These  requirements  and 
components  are  detailed  in  the  succeeding  descriptions  of  the  APEP  and  TREDS 
programs. 

The  Automated  Exercise  Preparation,  Evaluation  and  Preview  (APEP)  Tool 

The  need  for  preparing  and  responding  to  dynamic  exercise  challenges  must  be 
met  (Pemberton,  Classe,  Bradley,  &  Wilson,  1994).  To  overcome  the  “time  consuming 
and  labor  intensive”  (p.  6-13)  process  of  creating  training  scenarios,  the  Automated 
Exercise  Preparation,  Evaluation  and  Preview  (APEP)  tool  was  developed  by  the  Navy 
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(Pemberton,  et  al.,  1994).  The  APEP  tool  was  designed  to  alleviate  difficulties  associated 
with  exercise  management  and  reduce  errors  in  placement  of  forces  for  an  exercise. 

The  forces  who  participate  in  global  and  national  networked  simulation 
environments  have  multiple  constraints  and  requirements  to  meet  in  order  to  conduct  a 
large-scale  training  exercise.  APEP  is  a  prototype  designed  to  “reduce  exercise 
preparation  time  from  months/weeks  to  days/hours”  (Pemberton,  et.  al.,  1994,  p.  6-13). 

To  achieve  this  objective  APEP  has  three  main  capabilities: 

1 .  automated  exercise  entity  positioning  based  upon  a  specific  training  objective, 

2.  automated  platform  scripting  using  computer  generated  entities,  and 

3.  automated  association  of  training  objectives  with  performance  measurement 
criteria  (Pemberton,  et  al.,  1994,  p.  6-13). 

The  first  APEP  module  to  be  prototyped  is  the  Training  Exercise  Force  Laydown 
(TEFL).  The  TEFL  module  uses  expert  system  techniques  to  enable  a  training  supervisor 
to  develop  difficult  and  realistic  scenarios.  The  training  supervisor  specifies  the  general 
nature  of  the  training  (situation  training  exercises  or  events),  establishes  friendly  and 
enemy  forces  to  play  in  the  scenario  (task  organization  and  condition  assessment),  and 
chooses  tactical  constraints  and  limitations  for  battle.  After  this  information  is  entered, 
TEFL  automatically  initiates  exercise  starting  locations,  routes,  navigational  information, 
and  weapons  status  for  both  sides  (scenario  conditions  and  initialization  parameters). 
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The  second  module  developed  in  APEP  is  for  Distributed  Interactive  Simulation 
(DIS)  training  exercises.  As  an  interface  tool,  the  Automated  exercise  Distribution  and 
Display  (ADD)  electronically  mails  the  APEP  output  to  DIS  participants  using  the  DIS 
protocol  data  units  (Pemberton,  et  al.,  1994).  This  tool  will  alleviate  some  of  the  initial 
scenario  initialization  and  setup  efforts,  while  providing  an  interface  to  other  models 
under  the  DIS  exercise  simulation  architecture. 

The  APEP  program  has  sought  to  reduce  planning  and  preparation  time  for 
simulation-based  training  in  the  U.S.  Navy.  The  two  goals  of  the  APEP  tool  are  to 
coordinate  the  initial  laydown  structure  of  naval  forces  and  enable  faster  access  to  large 
amounts  of  data  for  the  DIS  simulation  exercise.  The  initial  prototype  of  the  tool  is  still 
being  evaluated,  but  its  development  has  been  accepted  enough  to  warrant  continued 
investigation  of  the  concept  for  training  plan  development.  The  U.S.  Army  is  in  a  similar 
position  with  its  prototype  for  automated  training  planning  of  simulation-based  training 

The  TRaining  Exercise  Development  System  (TREDS) 

Recognizing  that  future  battlefield  training  and  preparation  for  “other  than  war” 
missions  will  rely  more  and  more  on  simulators  and  simulations,  unit  commanders  must 
incorporate  new  ways  to  develop  effective  training  plans  (Crissey,  et  al.,  1994). 
Iimovative  methodologies  must  be  applied  to  the  planning  process  to  match  essential 
mission  training  tasks  against  constrained  training  resources.  The  TRaining  Exercise 
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Development  System  (TREDS)  is  a  prototype  system  being  engineered  by  the  author  to 
meet  these  needs  in  the  U.S.  Army. 

The  purpose  of  TREDS  is  to  provide  a  computer-based  system  to  minimiyft 
planning  time  and  maximize  the  available  training  time.  The  training  tasks,  allocation  of 
equipment  and  personnel,  and  the  degree  of  difficulty  for  exercise  conditions,  must  all  be 
considered.  Effective  plaiming  is  essential  for  all  of  these  factors  to  achieve  training 
mission  objectives. 

Anderson  (1987)  asserts  that  in  order  to  understand  human  cognition,  a  person 
needs  to  “understand  both  mental  algorithms  (procedures)  and  their  implementation”  (p. 
467).  The  procedures  and  their  implementation  in  the  Training  Exercise  Development 
System  (TREDS)  were  constructed  from  the  mental  models  of  SME’s  experiences  using  a 
cognitive  analysis  tool  which  develops  production  rules.  The  TREDS  program  is 
intended  to  provide  umt  commanders  with  the  knowledge  or  cognitive  requirements  to 
conduct  planning  for  simulation-based  training 

To  organize  the  training  planning  development  system  in  TREDS,  the  structure 
includes  four  modules  composed  of  training  analysis,  task  organization,  exercise  selection 
and  exercise  development.  As  the  TREDS  Systems  Engineer,  the  author  utilized  this 
study  to  assess  the  theory  that  automated  planning  tools  like  TREDS  have  the  capabilities 
to  reduce  planning  time,  enhance  training  plan  development,  and  provide  information  for 
future  traimng.  TREDS  is  used  in  this  study  as  the  computer-based  planning  tool  for 
simulation-based  training. 
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Replicating  the  Army  training  and  evaluation  process,  TREDS  has  a  computer- 
based  procedure  to  convert  training  results  (tactical  measures  of  effectiveness  (MOE)) 
into  proficiency  ratings.  This  process  begins  with  the  lowest  task  level  to  define  the 
standard  or  criterion  a  priori  for  pass/fail  (Go/No  Go)  tasks  in  training  At  this 
criterion/standard  level,  the  Army’s  current  training  assessment  system  resembles  the 
criterion-referenced  evaluation  system  described  by  Gonzalez  and  Ingraham  (1994).  The 
task  ratings  of  pass  or  fail  can  then  be  adjudicated  by  integrating  the  simulation  output  of 
measures  of  effectiveness  values.  The  ratings  are  finally  aggregated  through  the  subtask 
and  task  levels  and  rated  as  either  trained,  partially  trained  or  imtrained.  To  determine  the 
overall  training  success  and  effectiveness,  the  trainer  compares  and  assesses  the  collective 
team’s  proficiency  status  before  and  after  the  simulation. 

The  design  of  a  computer-based  planning  tool  requires  a  conceptual  framework 
upon  which  the  users  can  understand  the  process  being  modeled.  A  training  planning 
system  would  have  to  incorporate  the  recognizable  attributes  and  features  which  allow  the 
users  to  develop  a  mental  model  of  the  conceptual  system  itself.  Also,  a  training  planning 
system  must  contain  metaphors  and  analogies  which  will  convey  the  conceptual  system’s 
mechanics  and  operations  and  assist  the  novice  in  understanding  the  conceptual  training 
plan  development  process.  This  model  enhances  the  user’s  mental  picture  of  the  system’s 
components,  inputs  and  outputs,  or  overall  structure.  All  of  these  areas  are  important  for 
the  success  of  a  computer-based  planning  tool  to  enable  effective  simulation-based 
training  plan  development. 
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Besides  APEP  and  TREDS,  as  the  study  has  indicated  earlier,  there  are  no 
computer-based  planning  tools  capable  of  handling  the  requirements  for  planning 
simulation-based  training  exercises  and  events.  The  TREDS  design  which  enables 
computer-based  simulation-based  training  planning  was  developed  by  the  author  as  a 
response  to  the  military’s  need  for  automated  planning  of  simulation-based  training  The 
APEP  tool  developed  by  the  Navy  was  considered  as  a  potential  planning  tool.  However, 
the  APEP  tool  is  still  a  prototype  and  is  not  as  functional  as  the  TREDS  tool.  Due  to  its 
lack  of  functionality,  the  APEP  tool  was  not  considered  further.  This  study  will  now 
examine  the  measures  of  effectiveness  applicable  to  a  training  plan  development  tool. 

Measures  of  Training  Effectiveness 

Assessment  methods  for  training  planning  tools  are  needed  for  ensuring  that  the 
training  planning  process  meets  training  goals.  These  methods  correlate  directly  with  the 
factors  involved  in  adopting  new  training  strategies  which  use  new  training  media  such  as 
simulations.  Effectiveness  is  defined  by  Rouse  (1991)  as  “the  extent  to  which  the 
structure  leads  to  improved  performance,  makes  a  difficult  task  easier,  or  enables 
accomplishing  a  task  that  could  not  otherwise  be  accomplished”  (p.  23).  In  an  era  of 
continued  budget  reductions,  Djang,  Butler,  Laferriere,  and  Hughes  (1993)  reiterate  the 
need  for  effective  training  planning  and  simulation  integration: 

Training  developers  need  a  tool  to  assist  them  in  determining  how  to  mix 
existing,  new  and  proposed  training  systems  into  effective,  less  costly 
training  strategies.  The  tool  would  indicate  which  strategies  fall  within 
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budget  constraints,  what  the  tradeoffs  are  between  training  effectiveness 

and  cost...  (p.  1) 

While  there  are  a  variety  of  measures  and  assessment  tools  for  evaluating 
individual  or  team  performance,  the  appropriate  measures  must  be  selected  for  the 
traimng  situation.  The  majority  of  training  is  often  conducted  without  providing 
empirical  results  due  to  the  unfeasibility  of  collecting  data  from  an  incorrect  or 
inadeqiiate  training  model  design  (Sloman,  1994). 

In  classifying  measurement  approaches.  Rouse,  Cannon-Bowers,  and  Salas 
(1992)  identify  three  categories: 

1.  empirical  modeling, 

2.  analytical  modeling,  and 

3.  verbal/written  reports. 

This  study  will  utilize  the  anal)^ical  modeling  and  verbal/written  techniques  to 
assess  the  experimental  data  obtained  at  the  conclusion  of  testing.  First,  the  analytical 
measures  which  evaluate  effectiveness  of  training  performance  are  reviewed. 

Performance 

Plans  are  effective  as  long  as  they  improve  performance  in  the  workplace  and 
thereby  contribute  to  the  achievement  of  the  organization’s  mission  and  objectives. 
Performance  effectiveness  should  always  be  measured  in  quantitative  terms,  such  as, 
reduced  waste,  increased  sales,  shorter  production  time,  or  fewer  accidents  (Spurgin, 
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Moieni,  &  Orvis,  1993;  Fitz-enz,  1994).  Being  able  to  meet  training  goals  by  conducting 
the  tasks  correctly  is  a  result  of  having  an  effective  imderstanding  of  training  plan 
development 

Programmatic  Measures  of  Effectiveness  (PMOE)  assess  performance  changes  in 
training  and  operational  contexts  (Department  of  Defense,  1993).  Data  analysis  and 
measurement  of  meaningful  results  continue  to  be  an  area  for  improvement  for  industries 
such  as  the  nuclear  power  plants  (NPP)  (Spurgin,  Moieni,  &  Orvis,  1993).  One  key 
objective  is  to  identify  quantitative  assessments  of  training  performance  results. 

Searching  for  a  way  to  measure  performance  effectiveness.  Hall,  Dwyer,  Cannon- 
Bowers,  Salas,  and  Volpe  (1993)  developed  the  Tactical  Decision  Making  Under  Stress 
(TADMUS)  project.  TADMUS  utilizes  key  techniques  in  concept  mapping,  protocol 
analysis,  cognitive  task  analysis,  and  cognitive  networking  to  model  knowledge 
requirements  of  team  members  for  threat  response  management  and  situation  assessment. 
The  Anti-Air  Warfare  Teamwork  Observation  Measure  (ATOM)  evaluation  in  TADMUS 
measures  four  teamwork  dimensions:  situation  assessment,  communication, 
compensatory  behavior  and  team  coordination.  The  TADMUS  program  is  depicted  by 
the  author’s  notes  in  Table  4  from  an  interview  with  one  of  the  Navy’s  training  system 
designers.  Dr.  Joan  Johnston  (December  7, 1994). 
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TABLE  4 


TRAINING  ASSESSMENT  MATRIX  FOR  TADMUS 


Type  of  Training 

Individual 

Team 

Process 

Process  Analysis 

ATOM  (measures 
of  performance) 

Outcomes 

(standards) 

Rating  Scales,  Behavioral  Observations 

Team  Standard 
Performance 

The  TADMUS  and  ATOM  dimensions  represent  measurement  techniques  and 
organization  which  may  be  applicable  for  evaluating  simulator  training.  Expanding  the 
focus  of  this  study,  researchers  could  use  these  dimensions  for  training  simulation 
exercises  in  emergency  management  centers,  nuclear  power  plant  command  centers,  and 
other  military  operations  centers. 

One  area  for  improvement  in  the  training  development  process  is  the  organization 
of  tasks  and  standards  for  establishing  consistent  training  events  and  ratings  across  all 
units  (Hall,  et  al.,  1993).  The  quality  ratings  procedure  used  in  the  study  reflects  the 
quality  rating  process  proposed  by  Cooper  and  Harper  (1969)  to  quantitatively  measure 
effectiveness  on  a  consistent  basis. 

Cognitive  mapping  and  clustering  techniques  define  qualitative  measures  of  the 
training  planning  tool  (Gcijuoy  &.  Spitz,  1966;  Roenker,  Thompson,  &  Brown,  1971; 
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Frankel  &  Cole,  1971;  Naime,  1991).  The  literature  on  psychological  memory  recall 
concerns  techniques  such  as  position  memory  (Naime,  1991).  Canas,  Bajo,  and 
Gonzalvo  (1994)  state  that  “it  is  possible  to  assess  subjects’  mental  representation  or 
mental  models  using  tasks  such  as  recall,  categori2ation  or  relatedness  judgments”  (p. 
797).  This  study  examined  these  techniques  for  measuring  the  effectiveness  of  a 
computer-based  tool  to  enhance  a  person’s  mental  model  or  cognitive  mapping  of  the 
training  plan  development  process. 

Plans  are  efficient  if  they  make  people  effective  in  less  time  or  with  less  money 
than  it  takes  with  alternative  modes  of  getting  people  to  perform  (Parry,  1991).  In  other 
words,  efficiency  is  dependent  on  effectiveness.  It  is  a  waste  of  time  and  money  to  make 
an  ineffective  plan  more  efficient  (Parry,  1991).  Assuming  that  a  plan  is  effective,  the 
focus  will  now  be  on  the  efficiency  of  the  training  plan  development  measurements. 

Measures  of  Training  Efficiency 

In  simulation-based  training,  measurements  confirm  whether  trainers  have  created 
training  scenarios  to  exercise  the  subject’s  cognitive  skills  in  a  variety  of  situations  which 
encompass  all  major  procedures  (Roth,  Mumaw,  &  Pople,  1992).  The  effectiveness  of 
the  training  plan  must  first  be  confirmed  prior  to  proceeding  towards  improving  the 
training  plan’s  efficiency.  The  efficiency  of  a  system  is  the  ratio  of  output  of  benefits 
divided  by  the  input  of  resources.  Planning  and  strategies  impact  on  the  efficient  use  of 
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resources.  Two  of  the  most  important  aspects  of  efficiency  in  simulation-based  training 
pertain  to  costs,  and  the  scheduling  of  resources  and  persoimel. 

Cost 

In  addition  to  effectively  planning  and  conducting  simulation-based  training,  the 
costs  should  be  minimized  as  much  as  possible.  Costs  incorporate  time,  funding,  and 
resources.  Simulation  participants,  trainers  and  site  managers  must  ensure  that  the 
exercise  is  well-organized  and  tailored  to  the  training  audience  to  avoid  uimecessary 
expenditures.  For  some  simulation  exercises,  “the  most  sophisticated  [ones]  run  into  a 
number  of  man-years  and  several  hundred  thousand  dollars”  (Keiser  &  Seeler,  1987,  p. 
466).  In  the  military,  costs  for  large-scale  simulation  exercises  have  been  observed  by  the 
author  to  be  in  the  millions  of  dollars.  There  are  also  months  of  manpower  and  facility 
usage  devoted  to  these  exercises. 

Examining  the  expenses  of  training,  Sloman  (1994)  states  that  “information 
systems  on  training  are  vital  but  neglected”  (p.  141)  and  justifies  capturing  adequate 
training  information  for: 

1.  cost  control, 

2.  the  provision  of  information  for  decision  making  by  line  management,  and 

3.  the  provision  of  information  for  decision  making  by  the  training  manager. 
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Costs  are  not  always  measured  accurately  due  to  the  unknown  variables  which 
impact  upon  the  simulation-based  training  exercises.  To  measure  quality  and 
productivity  training  costs,  Smith  (1987)  provides  a  review  of  the  techniques  available. 
Some  of  these  techniques  highlighted  by  Smith  (1987)  and  Sloman  (1994)  are  related  to 
costs  in  terms  of  time  and  money.  The  measurements  are  volumes  of  training  activity 
and  the  costs  associated  with  training  (Sloman).  Participation  in  training  measured  “in 
terms  of  days  or  hours  of  training  received  is  the  most  accessible  training  metric” 
(Sloman,  p.  145).  For  off-the-job  training,  these  measurements  are  often  inaccurately 
measured  (Sloman). 

The  second  part  of  measuring  training  resources  is  the  monetary  costs  of  the 
training.  Sloman  (1994)  implies  that  costs  are  often  difficult  to  measure  due  to  the 
various  methods  organizations  use  in  applying  cost  factors  to  internal  training  programs. 
External  training  programs  are  straightforward  to  measure  costs,  but  the  internal  training 
cost  measurements  do  not  always  use  the  same  metrics  and  relationships  in  defining 
actual  charging  to  various  accovmts  (Sloman,  1994).  As  the  cost  of  this  study  is  weighed 
against  the  time  involved  in  using  the  system,  the  same  dilemma  will  likely  surface.  For 
this  study,  the  efficiency  cost  will  be  related  to  the  perceived  operator  workloads  and 
recorded  performance  times. 

One  of  the  final  considerations  for  cost  is  the  method  for  recording  training 
evaluations  and  data.  Tumage,  et  al.,  (1989)  note  that  with  the  move  towards  more 
simulation-based  training,  the  Navy  needs  to  look  for  better  ways  to  automate  data 
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collection  and  analysis  for  improving  the  efficiency  of  the  training  process.  This  aspect 
of  training  is  often  the  last  consideration  for  funding,  yet  it  yields  results  critical  to  future 
training  plan  development.  Equally  important  to  measurement  of  efficiency  as  cost  is,  the 
factor  of  scheduling  resources  and  personnel  for  simulation-based  training  warrants 
consideration. 


Scheduling 

To  organize  the  training  plan  in  order  to  meet  its  objectives,  the  efficiency  of  a 
schedule  must  also  be  measured  for  the  time  saved  and  output  of  training  gained. 

McCord  (1987)  states  that  a  schedule  is  essential  to  satisfactory  planning  and  preparations 
in  that  it: 

1 .  specifies  what  maj  or  topics  will  be  covered  with  the  trainee, 

2.  indicates  what  amoimt  of  time  will  be  devoted  to  each  major  topic,  and 

3.  assigns  responsibility  for  each  situation  (p.  377). 

In  scheduling  the  simulation-based  team  training,  various  time  considerations  are 
required.  The  process  involves  capturing  information  for  timing  of  events  and 
synchronization  of  related  tasks  on  a  schedule.  For  training  plan  development  with  a 
simulation,  Keiser  and  Seeler  (1987)  assert  that  the  process  is  more  complicated  than 
conventional  scheduling  in  that  “the  trainer  must  account  for  all  practical  contingencies, 
and  prepare  schedules,  space,  materials,  personnel,  and  responsibilities”  (p.  467). 
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Training  time  must  be  planned  and  scheduled  efficiently  with  respect  to  the 
complexity  and  availability  of  resources  and  the  variety  of  personnel  involved.  The 
measurement  of  training  efficiency  for  scheduling,  unlike  cost  considerations  discussed 
earlier,  is  more  qualitative  than  quantitative.  Cost  efficiency  is  a  measurable  quantity  in 
terms  of  workload  and  time  required  to  conduct  tasks  in  planning  simulation-based 
training. 


Summary 

Chapter  2  began  with  an  overview  of  the  training  process  including  the  training 
model,  individual  and  team  training,  simulation-based  training  and  the  Janus  simulation. 
This  section  also  reviewed  the  systematic  training  model,  types  of  training  in  teams, 
benefits  and  limitations  of  simulations. 

Also  included  was  a  literature  review  related  to  current  methods  for  training  plan 
development.  Present  manual  methods  are  widely  used,  but  new  methods  are  needed 
which  can  handle  the  intricacies  of  simulation-based  training.  To  learn  or  plan  faster  and 
easier,  there  is  the  potential  that  novice  trainers  may  be  assisted  through  training  plan 
development  mental  models. 

The  next  section  identified  several  computer-based  planning  tools  available  for 
general  planning,  training  planning  and  automated  planning.  The  literature  search  only 
yielded  several  automated  plaiming  tools.  Likewise,  for  specific  domains,  such  as  market 
strategies  and  training,  there  were  only  a  few  computer-based  tools  available. 
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A  final  section  focused  on  measures  of  training  effectiveness  and  efficiency. 
Performance  measurements  were  reviewed  to  determine  the  effectiveness  of  a  training 
plan.  Costs  and  schedules  were  considered  to  show  whether  a  training  plan  is  efficient. 
Table  5  summarizes  the  results  of  the  literature  review  with  the  systems  which  apply  to 
the  components  of  this  study,  clearly  showing  the  need  for  a  computer-based  planning 
tool  to  prepare  for  simulation-based  training. 

Planning  enables  effective  and  efficient  training  while  the  use  of  the  planning 
process  optimizes  training  time.  Computer-based  planning  tools  capture  subject  matter 
expert’s  (SME)  mental  models  within  the  framework  of  the  software  tool  to  convey  the 
complexities  and  functionalities  inherent  to  planning  for  simulation-based  training.  With 
less  opportunities  for  error  and  inefficient  resource  usage,  the  automated  tool  has  the 
potential  to  enhance  training  proficiency,  increase  imderstanding  of  the  training  planning 
process,  reduce  costs,  and  build  linkages  to  future  training. 

No  planning  tools  are  available  which  meet  this  need  for  simulation-based 
training.  Knowledge  and  experience  is  essential  for  the  training  planning  development 
process.  Possessing  limited  expertise,  novices  must  apply  the  correct  procedures  for 
conducting  simulation-based  training  to  ensure  effectiveness  and  efficiency.  An  essential 
requirement  for  simulation-based  training  is  the  development  of  a  tool  which  will 
automate  planning  tasks  in  the  context  of  the  plaiming  process  structure. 
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TABLES 


LITERATURE  REVIEW  MATRIX 
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As  discovered  from  a  review  of  the  literature  and  the  author’s  professional 
experience,  research  and  development  efforts  in  computer-based  planning  tools  to 


enhance  simulation-based  training  are  currently  very  limited.  The  scope  of  the  study 
would  be  too  exhaustive  if  the  study  included  all  of  the  simulation  users  in  academics, 
industry  and  government.  Therefore,  the  study  focuses  on  the  government,  and  is  further 
narrowed  to  the  U.S.  Army.  For  this  experiment,  the  U.S.  Army  provided  the  testing 
subjects  in  its  role  as  a  proponent  of  simulation-based  team  training. 

To  assess  the  hypothesis  that  computer-based  planning  tools  do  enhance  the 
effectiveness  and  efficiency  of  team  simulation-based  training  planning,  the  computer- 
based  planning  tool  used  for  this  study  is  the  Training  Exercise  Development  System 
(TREDS).  The  Janus  simiilation  is  utilized  for  team  simulation  training.  The  methods 
and  procedure  for  the  study  are  now  discussed  in  Chapter  3. 
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CHAPTERS 


METHODS  AND  PROCEDURE 

This  chapter  details  the  research  design,  data  collection,  data  analysis  methods 
and  procedures  used  in  this  study.  Test  subjects  were  from  two  U.S.  Army  locations: 

Fort  Knox,  Kentucky,  and  Fort  Hood,  Texas.  Research  data  were  collected  dxuing  a  week 
at  each  location.  To  maintain  consistency  among  the  two  groups,  a  realistic  training 
practical  exercise  was  developed  (see  Appendix  D).  The  study’s  main  purpose  was  to 
assess  the  ability  of  computer-based  planning  tools  for  improving  training  plan 
development  effectiveness  and  efficiency  in  simulation-based  team  training  exercises. 

Research  Design 

The  research  design  method  used  in  this  study  served  to  provide  insight  into  the 
main  question  raised  earlier—  What  contributions  do  computer-based  planning  tools  offer 
to  simulation-based  team  training?  To  answer  this  question,  three  sub-questions  were 
developed.  They  are: 

1 .  Will  a  computer-based  planning  tool  enhance  the  mental  model  of  the  training 
plan  development  process? 

2.  Are  more  effective  training  plans  created  and  results  achieved  with  computer- 
based  planning  tools? 
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3.  Is  simulation-based  training  planning  more  efficiently  accomplished  using 
computer-based  tools? 

Subjects 

The  study  was  conducted  using  U.S.  Army  personnel  who  plan  activities  and 
simulation-based  team  training  for  their  units.  Training  planning  is  normally  conducted 
by  the  leaders  of  the  organization  for  their  responsible  individuals  and  teams. 

The  test  population  of  the  experimental  study  included  officers  and 
noncommissioned  officers  in  U.S.  Army  Armor  battalions  in  two  separate  locations. 
Within  a  large  unit,  such  as  a  battalion  of  about  700  soldiers,  there  are  approximately  80 
key  leaders  involved  in  planning  training  exercises.  Designating  subjects  for  the  teams  to 
be  tested  was  accomplished  through  coordination  with  unit  leaders  at  Ft.  Knox  and  Ft. 
Hood.  Volimteers  from  this  population  included  platoon  leaders,  platoon  sergeants, 
training  sergeants  and  company  commanders. 

Test  subjects  were  familiar  with  simulations  and  planning  for  training  processes  in 
the  U.S.  Army.  Subjects  were  randomly  assigned  to  the  respective  experimental  design 
groups  in  two-person  teams  which  represented  a  standard  assignment  of  personnel  for 
planning  and  simulation  training. 

One  requirement  of  a  control-group  design  is  to  run  two  experiments 
simultaneously.  Each  location  required  at  least  one  month  to  set  up  and  organize  for  the 
TREDS  experimental  testing.  The  preparation  for  the  experiment  entailed  conducting 
several  onsite  visits  to  introduce,  anticipate,  and  coordinate  with  subjects  and  support 
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personnel  at  both  locations.  To  maximize  the  statistical  power  of  the  test  results,  the 
research  goal  was  to  have  at  least  30  two-person  teams  for  both  the  control  and  the 
experimental  groups.  For  the  test,  the  number  of  two-person  teams  which  completed  the 
experiment  were  25  in  the  manual  (control)  group  and  27  in  the  automated  (treatment) 
group. 


Experimental  Design 

The  selected  research  method  was  a  hybrid  arrangement  of  the  pretest-posttest 
control  group  design  and  the  posttest-only  control  design  depicted  in  Table  6. 


TABLE  6 

EXPERIMENTAL  DESIGN 


Random  Selection 

Observation  #1 

Treatment 

Observation  #2 

R 

Ot 

X 

O2 

Random  Selection 

Observation  #1 

No  Treatment 

Observation  #2 

R 

Oi 

O2 

The  treatment  group  used  the  computer-based  planning  tool  while  the  control 
group  conducted  training  planning  using  manual  planning  procedures.  The  treatment  and 
the  non-treatment  subjects  developed  a  training  plan  in  accordance  with  a  practical 
exercise  designed  for  the  study.  All  subjects  participated  in  all  aspects  of  the  experiment. 
The  selected  research  pretest-posttest  control  group  design  proposed  by  Gibson, 
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Ivancevich,  and  Donnelly  (1994)  is  “one  of  the  simplest  forms  of  true  experimentation 
used  in  the  study  of  human  behavior”  (p.  736).  However,  only  the  Adjusted  Ratio  of 
Clustering  (ARC)  dependent  variable  proposed  by  Geijouy  and  Spitz  (1966)  required  a 
pretest  and  posttest  to  determine  the  change  in  mental  model  development.  To  avoid 
interaction  effects,  Sproull  (1995)  asserts  that  “pretests  are  not  necessary  when  random 
assignment  is  used”  (p.  145).  The  other  dependent  variables  were  measvired  after  the 
experiment  or  in  phases  consisting  of  a  set  of  tasks.  Consequently,  the  actual 
experimental  design  was  a  combination  of  the  pretest-posttest  and  posttest-only  control 
group  model;  hereafter  referred  to  as  the  hybrid  control  group  design. 

Five  experimental  design  issues  considered  for  the  experiment  were: 

1.  history, 

2.  maturation, 

3.  homogeneity, 

4.  error  variance,  and 

5.  generalization. 

The  initial  two  research  design  factors  are  history  and  maturation.  For 
experimental  design,  Kellinger  (1986)  argues  that  these  are  the  two  most  significant 
factors  related  to  time.  History  represents  the  time  duration  of  the  experiment,  while 
maturation  captures  the  learning  which  occurs  during  the  experiment.  The  longer  the 
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period  between  the  pretest  and  posttest,  the  greater  the  chance  that  these  two  factors 
would  affect  the  study  (Kellinger,  1986).  For  this  study,  the  history,  or  time  between  pre¬ 
test  and  post-test,  was  approximately  one  day  for  each  subject.  The  short  duration  of  the 
experiment  did  not  require  a  need  to  investigate  the  maturity  effects.  Therefore,  it  is 
suggested  that  history  and  maturation  may  not  have  contributed  to  overall  results  due  to 
the  relatively  short  duration  of  the  experiment. 

A  third  research  design  consideration  involves  eliminating  imdesirable  variables 
from  the  experiment.  Homogeneity  is  one  method  for  ensuring  uniformity  of  variables, 
but  there  are  unknown  influences  which  are  not  easily  identified  and  can  cause  unwanted 
variance  (Kellinger,  1986).  The  best  way  for  controlling  extraneous  variance  is  to 
randomly  assign  subjects  to  experimental  groups  and  conditions,  as  well  as,  randomly 
assign  conditions  and  other  factors  to  experimental  groups  (Kellinger,  1986).  However, 
the  only  factor  or  condition  which  was  allowed  to  vary  was  the  use  or  nonuse  of  the 
TREDS  computer-based  plaiming  tool.  The  experiment  only  allowed  for  randomly 
assigned  subjects,  not  conditions.  The  single  condition  and  random  assignment  to  the 
one  factor  of  use  or  nonuse  of  TREDS  was  used  to  promote  homogeneity.  Table  7  shows 
the  homogeneity  of  the  two  groups.  Using  a  one-sample  t  test  of  individual  data  within 
the  two  groups,  overlapping  confidence  intervals  signify  that  there  was  no  significant 
differences  between  the  groups  in  four  demographic  areas.  The  conclusion  is  that 
homogeneity  appears  to  be  present  among  the  two  groups  in  the  areas  noted  on  Table  7. 
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TABLE? 


HOMOGENEITY  OF  GROUPS 


Variable 

Mean 

StdDev 

95% 

Confidence 

Interval 

Relative  PC 

Automated 

54 

3.259 

0.851 

(3.027,  3.492) 

experience 

Manual 

50 

2.860 

1.088 

(2.551,  3.169) 

Years  of  PC 

Automated 

54 

3.593 

1.281 

(3.243,  3.942) 

Experience 

Manual 

50 

3.200 

1.457 

(2.786,  3.614) 

Frequency  of 

Automated 

54 

3.000 

1.116 

(2.695,  3.305) 

PC  Use 

Manual 

50 

2.730 

1.166 

(2.399,  3.061) 

Training 

Automated 

53* 

2.321 

1.010 

(2.042,  2.599) 

Experience 

Manual 

49* 

2.286 

0.866 

(2.037,  2.535) 

Simulation 

Automated 

53* 

2.925 

1.158 

(2.605,  3.244) 

Experience 

Manual 

50 

2.740 

1.139 

(2.416,  3.064) 

*  Some  of  1 

lie  data  fields  were  no1 

t  completed  i 

by  the  subjects 

The  fourth  factor  for  good  research  design  is  minimization  of  error  variance  or 
variations  which  randomly  fluctuate  and  balance  out  to  zero  (Kellinger,  1986).  Individual 
differences  among  subjects  cause  error  variance,  while  errors  of  measurement  normally 
occur  due  to  “variation  of  responses  from  trial  to  trial,  guessing,  momentary  inattention, 
slight  temporary  fatigue  and  lapses  of  memory,  transient  emotional  states  of  subjects,  and 
so  on”  (Kellinger,  1986).  The  solution  proposed  by  Kellinger  (1986)  is  to  control  the 
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experimental  conditions  and  insure  good  reliability  by  accurately  assessing  measures  of 
effectiveness  or  scores.  A  large  error  variance  caused  by  imcontrolled  measurement 
errors  will  mask  systematic  variance  which  attempts  to  reveal  the  difference  between  the 
means.  To  assist  in  reducing  all  control  errors,  the  researcher  briefed  and  supervised  the 
subjects  on  the  testing  conditions  and  the  design  methodology. 

For  external  validity  of  data,  representativeness  or  the  generalization  of  results  to 
relate  to  a  specified  population  must  be  made  (Kellinger,  1986).  The  samples  should 
represent  their  population  while  the  experimental  environment  and  variables  examined 
must  be  realistic  and  true  to  the  purpose  of  the  study  and  its  conclusions  (Kellinger, 

1986).  To  satisfy  the  need  for  generalization,  the  study’s  experiment  was  conducted  with 
a  majority  of  persoimel  who  possessed  an  average  amount  of  training  experience  and 
were  familiar  with  simulations.  Table  7  shows  the  amoimt  of  training  experience 
between  the  two  groups  was  not  significantly  different  at  5%  level  of  significance.  The 
apportionment  of  computer  experience  abilities  between  the  mannal  and  automated 
planning  system  subjects  based  on  statistical  tests  indicates  that  the  subjects  in  both 
groups  were  similar.  The  training  experience  of  the  personnel  involved  was  not  the  only 
factor  considered  for  analysis.  Since  subject  teams  were  required  to  plan  for  and  execute 
a  computer  simulation,  simulation  experience  was  a  factor  in  assessing  the  team’s  ability 
to  plan  for  simulation-based  training.  Table  7  shows  that  the  simulation  experience  of  the 
subjects  varied  evenly  for  both  groups.  This  suggests  that  the  two  groups  contained 
similar  team  profiles  with  regards  to  using  simulations.  The  simulation  experiences  or 
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familiarity  of  the  subject  teams  therefore  had  little  effect  on  differences  between  each 
group’s  results. 

Not  only  does  the  research  need  to  randomly  assign  subjects  to  experimental 
groups  and  conditions,  but  the  power  of  the  experiment  needs  to  be  at  a  certain  level 
(Kellinger,  1986).  The  power  refers  to  the  ability  of  the  statistical  test  to  significantly 
detect  actual  differences  in  means  or  actual  improvements  due  to  a  training  planning  tool 
(Kellinger,  1986).  Based  on  the  personnel  and  resources  available,  the  experimental 
objective  initially  was  a  sample  size  of  at  least  30  teams  per  group.  This  objective  sought 
to  obtain  sufficient  power  for  discerning  differences  among  the  responses. 

The  final  design  consideration  involved  the  issue  of  generalization.  The  ability  of 
using  a  sample  group’s  experimental  results  for  application  to  the  population  is 
generalization.  The  test  audience  results  could  be  applied  to  the  population  at  large,  as 
long  as,  the  same  experimental  design  is  used.  Since  a  pretest  could  cause  either  the 
control  or  experimental  group  to  become  sensitized  to  the  experiment  and  strive  to  do 
better,  it  is  important  to  conduct  the  test  with  the  pretest.  The  result  could  be  a  lack  of 
generalization  for  applying  the  experimental  hypothesis  only  to  pretested  groups  versus 
the  population  at  large  as  originally  intended  (Kellinger,  1986). 

The  five  issues  presented  include  some  matters  of  concern  which  had  to  be 
addressed.  All  of  these  factors  were  heeded  to  ensure  the  study’s  experimental  design 
yielded  significant  and  consistent  results.  To  review  how  TREDS  relates  to  the 
experiment,  the  development  of  this  treatment  is  now  examined. 
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TREDS  Development 


The  Training  Exercise  Development  System  (TREDS),  a  computer-based 
planning  tool,  was  built  from  the  synthesis  of  mental  models  of  several  expert  planners. 
To  support  the  development  of  a  tool  to  automate  planning  for  simulation-based  trainings 
subject  matter  experts  (SME)  were  polled  for  descriptions  of  the  training  plan 
development  process.  In  addition  to  SME  interviews,  process  information  and  data  were 
gathered  from  informal  and  formal  conferences,  training  group  meetings  and 
demonstrations  of  initial  TREDS  prototypes.  The  experts’  conceptual  models  of  the 
training  planning  development  process  and  other  collected  design  parameters  were  then 
synthesized  into  the  final  system  design  for  TREDS.  The  ultimate  goal  of  this  effort  was 
to  construct  an  automated  tool  which  could  support  novice  planners  in  developing  their 
own  mental  model  of  the  training  plan  development  process  and  training  plans. 

An  experienced  training  SME  assisted  the  author  and  Dr.  Kent  Williams  in 
building  the  TREDS  model  with  cognitive  tasks  and  rules.  This  process  utilized  the 
Cognitive  Analysis  Tool®  developed  by  Dr.  Kent  Williams  at  the  University  of  Central 
Florida  as  a  means  to  organize  production  rules  for  mental  representation  of  processes 
(see  Appendix  E  for  the  rules).  The  resultant  TREDS  model  was  programmed  using  the 
FoxPro  database  in  a  Microsoft  Windows®  environment  which  applies  object  linking  and 
dynamic  data  exchange  for  transfer  of  data.  The  TREDS  program  runs  on  a  personal 
computer  with  at  least  eight  megabytes  of  random  access  memory,  200  megabytes  of  free 
hard  drive  memory,  Windows  for  Workgroups®  vs.  3.1 1  and  at  least  a  80486/50 
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megahertz  computer  processing  unit.  The  modular  structure  of  the  planning  tool  with  its 
training  plan  development  components  is  depicted  in  Table  8. 


TABLES 

COGNITIVE  ANALYTICAL  MODEL  OF  TREDS 


Events 

Tasks 

Scenarios 

Record  Mission 
Essential  Task 
List 

Assess  Training 
Proficiency 

Select  Training  Conditions 

Build  Calendars 
and  Schedules 

Prioritize  Tasks 

Select  Scenarios 

Allocate  Tasks  to 
Events 

Edit  Scenarios 

Check  Training  Aids  Devices, 
Simulators  and  Simulations 
Constraints 

Build  Input  Files  and  Exercise 
Products 

Capture  Assessments 

The  structure  and  components  of  the  initial  TREDS  model  were  subsequently 
compared  with  the  Training  Value  System  (TVS)  proposed  by  Fitz-enz  (1994).  The  TVS 
model,  outlined  in  Appendix  F,  served  as  a  basis  to  enhance  the  main  components  of  the 
TREDS  model.  Cognitive  conceptual  models  are  concerned  with  how  a  person 
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represents  information.  The  mental  model  presents  the  information  and  methodology  in  a 
logical  maimer  for  novices  to  use  the  expert’s  planning  process  with  minimal  training 
time. 

Representing  the  expert’s  mental  model  of  the  planning  process  for  simulation- 
based  training,  the  TREDS  planning  tool  was  then  ready  to  be  applied  to  a  military 
training  context.  For  other  domains,  the  TREDS  tool  can  be  tailored  to  the  applicable 
training  needs,  audiences,  priorities  and  characteristics.  The  development  of  a  training 
planning  process  incorporates  critical  elements  from  several  domains  which  use  training 
simulations.  These  simulation  systems  are  built  for  the  end-users  or  trainers  to  improve 
their  crews  and  overall  organizational  training  postures. 

The  TREDS  planning  tool  was  designed  as  a  bridge  to  connect  training  planning 
with  the  simulation-based  training.  The  simulation  itself  may  still  be  a  “black  box,”  but 
the  inputs  and  outputs  are  articulated  and  organized  through  TREDS  to  enhance  the 
training  experience.  Once  the  research  design  is  established,  the  next  step  is  the 
collection  of  experimental  data. 

Data  Collection 

For  instrumentation  of  the  experiment,  data  were  collected  to  determine  the 
pretest-posttest  adjusted  ratio  of  clustering  (ARC)  values  from  the  free  recall  supported 
with  feedback  from  the  position  memory  tests.  The  posttest-only  phase  of  the  experiment 
coiisisted  of  data  on  quality  ratings  (Cooper  &  Harper,  1969),  simulation  measures  of 
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effectiveness  (MOE)  (U.S.  Department  of  Defense,  1993),  task  load  index  (TLX)  scores 
(Hart  &  Staveland,  1988),  and  performance  times.  To  measure  backgrovmd  information 
and  solicit  post-test  feedback  on  structured  questionnaires,  the  author  developed  surveys 
similar  to  those  used  by  Stout  (1994).  Dr.  Kathy  Quinkert  at  the  Army  Research 
Institute,  Fort  Knox,  Kentucky,  assisted  the  author  in  modifying  the  questionnaires  to 
meet  the  study’s  objectives. 

Prior  to  the  experiments  at  both  Fort  BCnox  and  Fort  Hood,  a  pilot  test  in  Orlando 
served  to  organize  and  formulate  the  experiment.  For  the  pilot  test,  two  groups  of  eight 
subjects  who  were  learning  to  use  training  simulations  volimteered  to  assist  for  extra 
credit  in  a  military  science  course  at  the  University  of  Central  Florida  (UCF).  The  pilot 
test  provided  confirmation  and  areas  for  improvement  on  the  questionnaires  and  data 
collection  forms.  The  pilot  test  results  identified  potential  experimental  setup 
deficiencies,  enabling  improved  experimental  organization. 

One  additional  requirement  had  to  be  met  before  testing  could  begin.  The 
battalion  commander  at  Fort  Knox  requested  that  all  testing  material  be  approved  by  the 
U.S.  Army  Research  Institute’s  field  office  at  Fort  Knox  prior  to  the  experiment.  Dr. 
Kathy  Quinkert  provided  meaningful  feedback  to  improve  the  surveys  and  experimental 
layout.  As  a  result  of  this  process,  approval  was  granted  to  conduct  testing  in  May  1995 
at  Fort  Knox. 

The  testing  period  at  both  locations  (Fort  Knox  and  Fort  Hood)  required  one  week 
each.  Two-person  teams  were  composed  of  a  training  planning  officer  and  sergeant.  At 
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Fort  Knox  (Location  #1)  16  automated  and  15  manual  two-person  planning  teams  were 
tested  while  Fort  Hood  (Location  #2)  involved  1 1  automated  and  10  manual  two-person 
planning  teams.  The  total  number  of  teams  were  27  automated  and  25  manual  planning. 
Each  team  was  not  released  until  their  experimental  data  were  collected. 


Dependent  Variables 

Dependent  variables  were  selected  to  measure  the  effectiveness  and  efficiency  of 
the  responses  incurred  by  the  planning  tool  treatment.  The  following  dependent  variables 
are  discussed  with  regard  to  their  relationship  to  the  study’s  sub-hypotheses  (or 
hypothesis  points): 


1 .  Adjusted  Ratio  of  Clustering  (ARC) :  a  measure  of  the  ability  of  the  subjects  to 
organize  key  information  and  knowledge  requirements  in  the  training  plan 
development  process  for  simulation-based  training;  values  were  between  -1 .0 
and  +1.0  (posttest-pretest)  (Geijouy  &  Spitz,  1966). 

2.  Quality  Rating  Scores  (QRS):  the  overall  quality  or  effectiveness  of  the  tool  to 
assist  in  conducting  planning  for  simulation-based  training;  measure  used 
integer  rating  values  ranging  from  one  (the  best)  to  ten  (posttest  only)  (Cooper 
&  Harper,  1969). 

3.  Measure  of  Effectiveness  (MOE):ihQ  simulation  results  are  integers  based  on 
the  number  of  systems  varying  in  size  from  one  to  50  and  are  used  to  evaluate 
the  effectiveness  of  the  team-developed  simulation-based  training  plan 
(posttest  only)  (U.S.  Department  of  Defense,  1993). 

4.  Task  Load  Index  (TLX)  scores:  the  workload  measures  ranged  between  zero 
and  1 00  to  evaluate  the  workload  efficiency  of  the  planning  method  in  each  of 
the  four  phases  of  the  experiment  (posttest  only)  (Hart  &  Staveland,  1988). 
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5.  Performance  Times:  temporal  assessment  of  efficiency  (measured  in  minutes) 
for  conducting  the  required  tasks  for  training  plaiming  development  of 
simulation-based  training  (posttest  only). 

One  goal  of  the  experiment  was  to  measure  the  ability  to  improve  the  planner’s 
mental  image  of  the  training  plan  development  process.  Using  free  recall  and  position 
memory  techniques,  the  experimental  procedure  was  designed  to  allow  the  author  to 
investigate  the  ability  of  the  automated  planning  tool  to  convey  an  expert’s  mental  model 
of  the  training  plan  development  process  for  simulation-based  training.  The 
generalization  that  all  computer-based  planning  tools  enhance  simulation-based  training 
is  predicated  on  the  idea  that  the  tool’s  underlying  structure  is  similar  to  the  expert’s 
mental  model  of  the  training  plaiming  process. 

Adjusted  Ratio  of  Clustering  Values 

The  first  part  of  the  experiment’s  hypothesis  evaluated  the  improvement  of  the 
user’s  mental  model  through  the  use  of  a  computer-based  planning  tool.  Testing  the 
study’s  mental  model  hypothesis  involved  capturing  the  mental  model  of  the  training 
planner  before  and  after  using  a  computer-based  planning  program  or  current  manual 
planning  method.  Since  spatial  ability  is  a  key  component  of  assessing  an  individual’s 
capability  to  diagram  the  training  plan  development  process,  a  siuface  development 
cognitive  test  by  Ekstrom,  French,  Harman,  and  Dermen  (1976)  tested  this  factor.  Using 
the  ‘t’  test,  the  results  showed  that  there  were  no  significant  differences  between 
individuals  in  the  automated  and  manual  groups  (p  >  0.05).  The  95%  confidence 
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intervals  for  the  spatial  test  scores  are  between  1 7.40  and  41 .55  for  the  mannal  group  and 
between  20.63  and  35.40  for  the  automated  group.  Since  the  confidence  intervals 
overlap,  there  is  no  significant  differences  in  the  two  groups’  spatial  abilities. 

The  cognitive  psychology  tests  for  free  recall  and  position  memory  were  used  to 
evaluate  the  individual  mental  model  improvement  by  both  groups.  These  tests  have 
been  used  for  psychological  experiments  involving  memory  recall  of  lists  (telephone 
interview  with  Dr.  James  Naime,  January  1995).  Staimey  and  Salvendy  (1994)  also  used 
clustering  in  firee  recall  procedures  “to  examine  how  structuring  demands  influence 
memory  knowledge  structures  and  computer  performance  of  Field  Dependent  (FD) 
individuals”  (p.  602).  The  free  recall  and  position  memory  methods  provided 
quantitative  results.  These  data  are  measures  to  assess  whether  the  planning  method 
improves  the  user’s  mental  model  of  the  training  plan  development  system. 

The  free  recall  survey  requires  that  the  subject  describe  the  training  planning 
process.  The  clustering  of  key  terms  around  the  conceptual  information  model  for 
training  plan  development  are  compared  with  the  subject  matter  expert’s  (SME)  mental 
model  depicted  in  Table  8.  Geijouy  and  Spitz  (1966)  introduced  the  Adjusted  Ratio  of 
Clustering  (ARC)  for  quantifying  a  subject’s  ability  to  remember  categorical  items  in 
memory  recall.  Staimey  and  Salvendy  (1994)  state  that  “clustering  is  thus  seen  as  a 
consequence  of  organization  in  memory  of  the  target  information”  (p.  602).  For  the 
experiment,  the  knowledge  of  the  training  planners  is  the  result  of  memory  organization. 
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In  position  memory  tests,  a  similar  approach  provides  measures  to  check  the 
congruency  between  the  categorization  of  key  terms  in  the  training  plan  development 
model  (Table  8).  Naime  (1991)  provides  additional  guidance  and  examples  which 
describe  how  to  measure  position  memory  recall  abilities.  For  this  experiment,  the  free 
recall  and  position  memory  forms  are  in  Appendix  G,  while  the  ARC  values  for  free 
recall  are  in  Appendix  H.  The  position  memory  data  (in  Appendix  I)  were  used  to 
support  the  free  recall  results  and  conclusions. 

Quality  Ratings 

A  method  developed  by  Cooper  and  Harper  (1969)  for  measuring  pilot  aircraft 
ratings  was  also  used  as  a  means  to  gather  objective  scores  on  qualitative  attributes  based 
on  human  judgment  (Werewinke,  1974).  According  to  White  (1994),  the  “Cooper- 
Harper  Handling  Qualities  Rating  (HQR)  provides  a  subjective  measure  of  aircraft 
handling  qualities  and  piloting  workload  which  takes  into  accoimt  the  task  performance 
and  any  pilot  compensation  required  to  achieve  it”  (p.  6-6).  Instead  of  aircraft  handling, 
the  form  was  modified  to  ascertain  the  handling  characteristics  or  effectiveness  of  the 
planning  method,  manual  or  automated.  The  resultant  dependent  variable  was  a 
subjective  rating  of  the  effectiveness  of  the  automated  planning  process  versus  the 
manual  process. 

Applying  the  same  rating  convention  as  Cooper  and  Harper,  the  study’s  quality 
rating  varied  from  one  to  ten,  with  one  indicating  that  the  handling  quality  of  the  program 
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is  satisfactory.  Higher  values  show  less  effective  methods  with  increased  workload  and 
substandard  performance.  Appendix  J  shows  the  quality  rating  form  modified  for  this 
study.  Appendix  K  contains  the  quality  rating  experimental  data. 

Measures  of  Effectiveness. 

For  a  simulation  exercise,  the  measures  of  effectiveness  (MOE)  identify  the 
efficiency  or  performance  gains  of  a  training  program.  Simulation  data  are  converted 
directly  or  transformed  through  a  combination  of  output  variables  to  build  measures  of 
certain  aspects  of  the  training  plan,  such  as,  survivability  and  lethality.  The  Janus 
simulation  provides  output  data  for  quantifying  survivability  and  lethality  data,  such  as, 
the  number  of  combat  systems  which  were  destroyed  in  a  simulation  run.  This  research 
will  apply  MOE  to  evaluate  the  effectiveness  of  the  computer-based  planning  tool  to 
create  a  plan  replicating  a  realistic  scenario  for  a  military  combat  team  simulation  training 
exercise.  The  MOE  are  defined  in  Appendix  L  and  the  data  which  were  used  to  assess 
training  plan  effectiveness  are  in  Appendix  M. 

Task  Load  Index  Scores 

The  Task  Load  Index  (TLX)  is  a  multidimensional  rating  scale  procedure  that 
measures  changes  in  operator  workload  levels  for  different  sets  of  tasks  (Hart  & 
Staveland,  1988).  The  overall  workload  scores  include  six  bipolar  dimensional  ratings  of 
mental  demands,  physical  demands,  temporal  demands,  performance,  effort,  and 
fhistration.  This  methodology  was  developed  through  extensive  research  and 
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psychometric  analyses  in  laboratory  experiments  and  simulated  flight  experiments 
conducted  by  Hart  and  her  research  partners  (Nygren,  1991). 

Hart  and  Staveland  (1988)  state  that  the  purpose  of  their  research  was  to  develop  a 
workload  rating  scale  that  provides  a  sensitive  summary  of  workload  variations  within 
and  between  tasks  that  is  diagnostic  with  respect  to  the  sources  of  workload  and  relatively 
insensitive  to  individual  differences.  Nygren  (1991)  concludes  that  the  TLX  method  is 
very  beneficial  and  robust  in  measuring  perceived  workload  in  many  environments.  To 
obtain  a  quantitative  measurement  of  the  workload  for  TREDS  tasks  in  this  experiment, 
the  TLX  scores  were  appropriate.  The  TLX  scores  measured  the  effectiveness  of  the  two 
planning  methods.  The  forms  are  shown  in  Appendix  N  and  data  are  included  in 
Appendix  O. 

Performance  Times 

The  performance  time  for  a  set  of  tasks  was  recorded  (to  the  nearest  minute)  to 
measure  whether  or  not  the  experimental  group  conducted  the  experiment  faster.  The 
times  were  measured  in  foxrr  incremental  phases  of  the  experiment  where  it  was  necessary 
to  begin  another  more  unique  set  of  tasks  (form  in  Appendix  P).  The  resultant  times  for 
each  phase  were  compared  as  team  statistics  (see  Appendix  Q). 

Questionnaires 

To  augment  the  research  data  for  in-depth  analysis,  verbal  and  written  reports 
assist  m  measuring  efficiency  and  effectiveness  that  is  not  captured  through  empirical 
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evidence  (Rouse,  Cannon-Bowers,  &  Salas,  1992).  Structured  questionnaires  were  used 
to  gather  background  subject  information  on  areas  such  as,  military  training  experience, 
computer  experience  and  simulation  experience  (see  Appendix  R).  Open-ended  response 
questions  were  used  to  solicit  experimental  feedback  and  assessment  of  the  planning 
methods  (see  Appendix  S). 

Test  Procedure 

Anderson  (1987)  suggests  that  once  the  mental  mechanisms  which  implement  a 
task  are  computer-based,  the  next  step  is  to  estimate  parameters  of  its  implementation  for 
the  purpose  of  proving  assumptions  of  the  model’s  utility.  A  practical  exercise  was  used 
to  enable  replication  of  the  tasks  for  users  to  conduct  the  training  planning  process,  either 
manually  or  computer-based.  The  main  tasks  are  as  follows  with  more  detailed 
objectives  described  in  the  Practical  Exercise  packet  (Appendix  D): 

1 .  Solicit  demographic  information  from  subjects. 

2.  Request  individual  subjects  to  depict  a  mental  model  of  the  planning  process 
for  simulation-based  training  using  free  recall  and  position  memory 
techniques. 

3.  Capture  subject  feedback  through  time  sheets,  task  performance  surveys  and 
comparative  evaluations  during  the  training  planning  process  conducted  in  the 
steps  below: 

a.  Conduct  a  task  analysis  for  simulation-based  training. 

b.  Assess  training  proficiency  and  task  prioritization. 
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c.  Create  and  edit  simulation-based  training  scenarios  to  build  a 
training  plan. 

d.  Evaluate  and  rehearse  Janus  simulation  scenario  using  the 
developed  plan. 

e.  Run  the  Janus  simulation  scenario. 

f.  Assess  training  proficiency  based  on  Janus  simulation  results. 

4.  Repeat  step  #2. 

5.  Solicit  verbal  and  written  freeform  feedback  from  subjects. 

These  steps  enabled  the  collection  of  data  and  information  representative  of  an 
actual  training  planning  situation  that  leaders  would  normally  encounter  in  conducting 
planning  for  simulation-based  training. 

Data  Analysis 

The  data  used  for  the  experiment  were  collected  using  forms  and  questionnaires 
from  related  work  in  the  fields  of  training,  planning,  simulation  and  cognitive 
psychology.  The  independent  variable  was  the  use  or  nonuse  of  the  treatment  tool, 
TREDS,  while  the  dependent  variables  were  the  adjusted  ratios  of  clustering  (ARC), 
quality  ratings,  task  load  index  (TLX)  scores,  performance  times  and  measures  of 
effectiveness.  Additional  background  information  obtained  from  the  demographics  and 
post-exercise  questionnaires  included: 

1 .  computer  and  Microsoft  Windows©  operating  system  familiarity, 
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2.  training,  planning  and  simulation  experience,  and 


3 .  self-assessment  of  knowledge  increase. 

Statistical  Methods 

In  seeking  the  appropriate  nonparametrical  test  for  the  comparison  between  the 
two  independent  and  randomly  selected  groups,  the  Maim-Whitney  test  was  chosen  since 
it  is  “the  usual  parametric  counterpart”  for  the  two-sample  t  test  (Conover,  1980). 
Additionally,  the  strength  of  the  Mann- Whitney  test  is  almost  as  strong  as  the  parametric 
t  test  even  if  normality  existed  (Conover,  1980).  In  some  cases,  data  could  produce  a 
distribution  that  is  difficult  to  identify,  as  well  as,  ciutail  the  number  of  data  points.  The 
unidentifiable  data  distribution  also  confirmed  the  rationale  for  using  nonparametrical 
statistics. 

The  experimental  data  showed  whether  or  not  there  is  significant  difference 
between  the  treated  and  the  control  groups.  The  use  of  TREDS  is  the  treatment  variable. 
If  there  were  any  significance  noted  by  the  nonparametrical  test  for  each  measure,  then 
there  is  a  difference  between  the  group  which  used  the  computer-based  planning  training 
tool  and  the  group  which  relied  on  manual/traditional  training  planning  methods. 

Summary 

The  methods  and  procedure  for  the  experiment  followed  lab  research  testing 
practices.  The  subjects  were  two-person  teams  of  U.S.  Army  training  planners  who  had 
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volunteered  their  time  to  test  the  hypothesis  of  computer-based  planning  tools  for 
simulations.  The  subjects  at  both  test  sites  used  the  planning  tools  and  Janus  simulation 
as  part  of  the  experiment. 

This  experimental  design  focused  on  the  use  of  data  to  show  the  difference 
between  those  who  use  and  those  who  do  not  use  a  computer-based  planning  tool.  The 
control  group  design  facilitated  gathering  information  on  both  mental  model  achievement 
and  comparison  of  the  two  methods  for  planning.  The  study’s  objective  was  to  assess  the 
hypothesis  that  computer-based  planning  tools  enhance  the  effectiveness  and  efficiency 
of  the  Janus  simulation-based  training.  The  next  chapter  discusses  and  analyzes  the 
results  collected  from  the  research  methodology  and  design. 
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CHAPTER  4 


DATA  ANALYSIS 

This  chapter  provides  the  analysis  of  the  data  for  the  study.  Analysis  was  based 
on  data  collected  from  25  manual  and  27  computer-based  planning  teams.  The  response 
measures  affected  by  the  independent  variable  were  the  adjusted  ratio  of  clustering 
(ARC)  values,  qtiality  ratings,  simulation  measures  of  effectiveness  (MOE),  task  load 
index  (TLX)  scores,  and  performance  times.  Demographic  information  and  post-exercise 
feedback  aided  the  interpretation  of  the  statistical  results. 

The  study’s  pretest-posttest  data  were  analyzed  using  the  Wilcoxon  signed  ranks 
test,  while  the  Mann-Whimey  test  was  used  for  assessing  the  other  four  response 
variables.  Histograms  also  depict  the  frequency  distributions  of  the  data.  The 
demographics  and  post-experiment  questionnaires  provided  insights  on  the  subjects’ 
assessments  of  the  planning  tools. 

According  to  Sproull  (1995),  “the  most  commonly  used  significance  level  is  .05” 
(p.  59).  This  study  seeks  to  minimize  the  risk  of  overstating  its  assertions  since  the 
research  is  important  to  the  body  of  knowledge  and  represents  doctoral-level  research. 
Thus,  the  level  of  significance  for  all  of  the  tests  was  set  at  5%.  The  results  will  now  be 
discussed  in  more  detail  as  to  determine  the  proof  of  the  hypothesis  at  a  level  of  5%. 
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Mental  Model  Measurements 


Hypothesis  Point  #1 :  A  computer-based  planning  tool  will  enhance  the  mental 
model  of  the  training  plan  development  process.  The  first  point  of  the  hypothesis  was 
tested  using  the  Wilcoxon  signed-ranks  test  This  test  compared  the  pretest-posttest 
results  for  both  groups.  The  fi’ee  recall  results  indicate  that  a  computer-based  planning 
tool  does  improve  training  planning  (Appendix  I). 

Initial  analysis  of  the  experiment  revealed  a  large  variance  in  both  the  pretest  and 
the  posttest  data  for  the  ARC  values.  A  large  index  of  variation  (TV)  ratio*  implies  that 
the  mean  would  not  be  a  good  estimation  of  the  true  average  of  the  population  (Kan, 
1995). 


TABLE  9 

ARC  VALUES  FOR  IV  COMPARISON 


Pretest 

Posttest 

Measure 

Manual 

Computer-based 

Manual 

Computer-based 

Mean 

-0.12 

-0.16 

-0.02 

0.10 

StDev 

0.35 

0.33 

0.29 

0.41 

rv 

2.36 

3.35* 

13.82* 

2.25 

*  Index  of  Variation  exceeds  3.0;indicates  that  variance  is  too  large  and  sample  size 

should  be  larger 


‘  The  formula  for  the  index  of  variation  is  (standard  deviation)  /  (mean). 
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Table  9  presents  the  IV  ratios  to  illustrate  the  magnitude  of  the  standard  deviation 
versus  the  mean.  The  standard  deviations  are  exceedingly  higher  than  their  respective 
means.  There  is  too  much  variance  to  conduct  parametric  statistics  on.  Additionally,  the 
plotting  of  a  stem-and-leaf  diagram  revealed  that  the  data  may  not  be  normally 
distributed.  Figures  2  and  3  also  indicate  such  a  trend  in  the  ARC  values.  According  to 
Milton  and  Arnold  (1986),  the  results  of  a  chi-sqiwe  test  on  data  with  large  variances  do 
not  always  validly  show  normality.  Milton  and  Arnold  assert  that  instead  of  using  a 
possibly  invalid  assumption  of  normality,  nonparametric  methods  are  “usually  superior 
for  analyzing  data  when  the  normal  theory  assumptions  are  not  met;  they  compare  very 
favorably  to  the  normal  theory  test  even  when  the  normal  theory  assumptions  are  met”  (p. 
242).  Therefore,  the  safest  statistical  method,  a  nonparametric  test,  will  be  used 
throughout  this  study  to  take  advantage  of  this  situation  in  case  there  is  not  enough 
normality  in  the  data. 
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Figure  2.  ARC  Value  Test  Results  for  the  Manual  Group 
Figure  2  shows  the  ARC  values  for  the  manual  groups  on  both  the  pretests  and  the 
posttests.  The  manual  planning  teams  did  not  appear  to  show  any  significant  gains  in  the 
subjects’  mental  models  of  the  training  planning  process.  Noting  the  missing  data  on  the 
right-side  of  the  graph  in  Figure  2,  one  can  observe  that  none  of  the  manual  subjects 
maximized  the  score  on  the  pretest.  The  manual  group  did  have  one  subject  who  was 
able  to  maximize  the  posttest. 

Figure  3  also  portrays  that  the  ARC  test  results  imply  the  potential  of  improved 
mental  models  for  the  computer-based  planning  groups  between  the  pretest  and  the 
posttest.  In  reference  to  some  of  the  apparent  missing  data  on  the  graph  in  Figure  3,  none 
of  the  automated  subjects  maximized  the  pretest  or  posttest,  while  there  were  also  no 
scores  of  0.78  in  the  prettest  results. 
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Figure  3.  ARC  Value  Test  Results  for  the  Automated  Group 
For  statistical  analysis,  the  Wilcoxon  signed-ranks  test  was  used  since  the  pretest 
and  posttest  ARC  values  were  related  variables  (see  Table  10).  At  the  significance  level 
of  5%  for  the  computer-based  planning  team  subjects  showed  that  they  gained  a 
significant  increase  in  the  understanding  of  the  training  planning  conceptual  model.  The 
confidence  intervals  between  the  two  tests  did  not  overlap,  showing  there  is  significant 
difference  between  the  pretest  and  the  posttest  scores  for  the  automated  group.  However, 
the  changes  in  ARC  values  from  pretest  to  posttest  for  the  manual  group  subjects  were 
insignificant  (p  >  0.05)  (Table  10). 
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TABLE  10 


PRETEST  VS.  POSTTEST  ARC  VALUES 


Variable 

Medianafte,  -  Medianbefore 

P-value 

Significance 

Manual 

.00 

0.362 

p>.05 

Automated 

.22 

0.001 

p<.05 

The  results  of  the  pretest-posttest  data  on  free  recall  suggest  that  the  computer- 
based  planning  tool  did  improve  the  mental  model  of  the  computer-based  team  members, 
while  there  was  no  significant  change  in  mental  models  for  the  manual  teams.  A 
computer-based  planning  tool  will  enhance  the  mental  model  of  the  training  plan 
development  process  with  a  probability  greater  than  95%  For  hypothesis  point  #1,  the 
mental  model  of  the  automated  teams  improved  at  the  5%  level  of  significance.  A  similar 
statistical  test  using  the  position  memory  data  (in  Appendix  I)  also  revealed  that  the 
automated  teams  improved  their  mental  models  (p  <  0.032)  while  the  manual  teams  did 
not  (p  <  0.965). 

To  examine  whether  there  was  a  difference  between  the  mental  models  of  the  two 
groups,  the  nonparametric  Mann- Whitney  test  was  conducted  with  results  shown  in  Table 
11.  The  first  line  of  Table  1 1  divulges  that  the  pretest  ARC  values  were  not  significantly 
different  for  both  groups,  implying  that  both  groups  started  the  exercise  with  similar 
mental  models.  For  the  free  recall  posttest  data  results,  however,  the  computer-based 
teams  showed  significantly  improved  mental  models  than  the  manual  planning  teams  had 
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(p  <  0.0389).  Analysis  of  the  position  memory  data  (in  Appendix  I)  also  refuted  the  free 
recall  results  by  showing  that  the  automated  teams’  mental  models  were  not  significantly 
more  accurate  in  the  posttest  results  (p<0.71 1)  versus  those  of  the  manual  teams.  This 
result  implies  that  the  position  memory  may  not  be  accurate  for  the  measurement  of 
mental  model  recall.  The  essential  measure  of  this  point  in  the  hypothesis  though  is  the 
free  recall  ARC  value  results. 


TABLE  11 

ARC  VALUES  BY  PRETEST-POSTTEST 


Variable 

Median,„to  -  Median„,g 

P-value 

Significance 

ARC  value  Pretest 

.00 

0.3449 

p>.05 

ARC  value  Pastiest 

.22 

0.0389 

p<.05 

Results  were  significantly  better  for  the  automated  planners  than  for  those  who 
applied  manual  planning  procedures.  These  free  recall  results  confirm  the  hypothesis  that 
the  automated  planning  tool  enhanced  the  mental  model  of  subjects  who  used  the 
computer-based  planning  tool.  Combined  with  the  pretest-posttest  analysis  already 
conducted,  the  hypothesis  point  #1  is  further  supported  at  the  5%  level  of  significance. 
Therefore,  the  data  show  that  the  computer-based  planning  subjects  improved  their 
mental  models  of  the  training  plan  development  process. 

Using  the  pretest-posttest  design,  it  was  confirmed  that  the  TREDS  computer- 
based  planning  tool  enhances  the  effectiveness  of  planning  for  simulation-based  training 
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by  improving  the  user’s  mental  model  of  the  training  planning  process.  The  next  point  of 
the  hypothesis  concerns  the  effectiveness  of  simulation-based  training  plans  created  by 
automated  planning  tools. 


Effectiveness  Measurements 

Hypothesis  Point  #2:  More  effective  training  plans  are  created  and  results 
achieved  with  computer-based  planning  tools.  The  two  dependent  variables  used  to  test 
for  significance  of  the  second  point  of  the  hypothesis  are  quality  ratings  and  simulation 
measures  of  effectiveness  (MOE). 

Quality  Ratings 

The  purpose  of  the  quality  ratings  was  to  gain  user  feedback  in  quantitative  terms 
on  the  ability  of  the  planning  method  to  assist  the  subject  in  building  effective  training 
plans.  Quality  ratings  were  elicited  in  integers  from  one  (the  best)  to  ten.  The  quality 
rating  results  were  significantly  better  for  the  automated  group  than  for  the  manual  group 
(see  Figure  4). 
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Figure  4.  Planning  Tool  Quality  Ratings 


To  determine  whether  there  was  a  significant  difference  between  the  groups  as 
suggested  by  Figure  4,  a  Mann- Whitney  nonparametric  t  test  was  conducted.  The  results 
of  the  nonparametric  test  on  the  posttest-only  control  group  design  are  depicted  in  Table 
12  (quality  rating  data  is  in  Appendix  K). 


TABLE  12 

QUALITY  RATING  STATISTICS 


Median,„to  -  Median^jn 

P-value 

Significance 

-1.5 

0.0002 

p<  .05 
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The  quality  rating  data  were  analyzed  to  determine  whether  the  computer-based 
teams  considered  the  training  planning  process  more  effective  than  the  manual  teams  did. 
The  median  team  Quality  Rating  scores  on  a  scale  of  one  to  ten  (with  one  being  the 
highest)  was  4.0  for  the  manual  group  while  the  computer-based  group  had  a  median 
score  of  2.5.  The  means  and  standard  deviations  (stdev)  of  the  Quality  Rating  scores  for 
the  manual  and  the  automated  teams  were  (meann,anuai  =  4.120,  stdev^anuai  =  1-394)  and 
(meanautornated  =  2.642,  stdeVamo^ated  =  1.047).  The  quality  rating  results  significantly 
show  that  the  computer-based  planning  tool  was  easier  to  use  and  more  effective  in 
planning  training  than  the  manual  method. 

The  quality  rating  scores  for  conducting  the  training  planning  were  measured  to 
ascertain  whether  the  effectiveness  of  the  planning  method  enabled  the  subjects  to 
perform  their  goals  with  ease.  Hypothesis  point  #2  also  sought  to  test  effectiveness  by 
examining  the  simulation’s  outcome  attained  by  the  teams’  training  plans. 

Simulation  Measures  of  Effectiveness 

To  assist  in  measuring  training  plan  effectiveness  for  the  posttest-only  control 
group  design,  the  simulation  measures  of  effectiveness  (MOE)  were  collected.  Analysis 
of  the  MOE  for  the  simulation  runs  sought  to  determine  the  difference  between  the  data 
from  the  training  plans  that  each  team  developed  for  the  Janus  training  simulation. 

Two  standard  military  MOE  are  the  number  of  friendly  forces  who  survive  (often 
focusing  on  critical  military  vehicles  such  as  tanks)  and  the  number  of  enemy  forces 
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destroyed  (see  Table  13).  These  two  measures  assist  in  determining  whether  an  iteration 
of  a  battle  simulation  is  successful.  For  civilian  emergency  management  simulations 
these  MOE  may  be  similar,  for  example,  with  regard  to  the  number  of  human  lives  saved 
and  amoimt  of  property  damaged. 


TABLE  13 


SIMULATION  MOE 


MOE#l 

MOE  #2 

MOE  #3 

#  Tanks  surviving 

#  Infantry  Vehicles 
surviving 

#  Enemy  Forces  Destroyed 

For  the  data  on  the  number  of  j&iendly  forces  which  survived  the  battle 
(survivability),  there  were  no  significant  differences.  At  a  significance  level  of  10.36%, 
the  manual  group  had  more  infantry  vehicles  remaining.  This  value  does  not  meet  the 
significance  level  of  5%.  The  lethality  MOE  which  showed  number  of  enemy  forces 
destroyed  by  the  teams  was  MOE  #3 .  From  the  graphical  comparison  between  the 
manual  and  automated  teams,  it  appears  that  the  results  are  similar  (see  Figure  5).  There 
was  no  significance  between  the  two  groups  in  terms  of  the  lethality  MOE  results  (p  > 
0.05). 
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Figure  5.  Simulation  MOE  #3  Results 

To  test  the  hypothesis  for  effectiveness,  the  Mann- Whitney  test  was  conducted. 
Measure  of  effectiveness  #3  was  not  significant  at  9.6%  for  the  computer-based  group 
with  a  median  of  44  enemy  ground  forces  destroyed  for  the  manual  group  versus  47 
enemy  ground  forces  destroyed  by  the  computer-based  group  (see  Table  14).  The  means 
and  standard  deviations  are  (meann,a„ua,  =  43.00,  stdev^anuaj  =  6.82)  and  (meanautomated  = 
44.91,  stdeVauto„,ated  =  8.81). 
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TABLE  14 


SIMULATION  MOE  #3  STATISTICS 


Variable 

Median^uto  -  Mediann,a„ 

P-value 

Sim.  MOE  m 

3 

0.096 

The  survivability  and  lethality  MOE  appear  to  be  contradictory  even  though  there 
is  no  significant  difference  between  the  results  for  the  two  groups.  It  coxild  be  possible 
that  the  teams  which  lost  more  systems  were  bolder  and  engaged  in  heavier  fighting,  thus 
incurring  more  enemy  losses.  Further  testing  using  other  MOE  may  dissolve  the  apparent 
presence  of  interaction  effects  between  the  two  MOE  variables. 

Since  only  one  of  the  two  measures  for  effectiveness  showed  significant 
differences  at  the  5%  level,  the  hypothesis  point  concerning  the  ability  of  computer-based 
planning  tools  to  improve  the  effectiveness  of  simulation-based  training  is  not  sufficiently 
proven.  These  results  imply  that  there  may  not  be  a  correlation  between  the  training  plan 
development  method  and  the  results  of  the  simulation  scenario.  Additional  research  in 
the  testing  of  simulation  plans  is  needed  to  justify  further  use  of  this  variable  as  one 
which  is  affected  by  the  planning  method. 

Efficiency  Measurements 

In  addition  to  the  effectiveness  of  assisting  the  user  in  understanding  the  system, 
the  time  and  workload  efficiency  of  the  training  planning  development  process  is  also 
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important.  The  third  point  of  the  hypothesis  states  that  simulation-based  training 
planning  is  more  efficiently  accomplished  using  computer-based  tools. 

To  measure  efficiency  for  the  posttest-only  control  group  design.  Task  Load  Index 
(TLX)  scores  and  performance  time  measures  were  collected.  Both  variables  required 
measurements  throughout  the  experiment.  The  experiment  was  divided  into  four  sessions 
as  follows: 

1 .  task  identification  and  prioritization, 

2.  training  plan  development, 

3.  simulation  rehearsal  and  execution,  and 

4.  performance  assessment. 

To  test  this  the  efficiency  point  of  the  study’s  hypothesis,  the  Task  Load  Index 
(TLX)  results  were  examined  to  show  whether  workload  was  reduced  for  computer-based 
planning  of  simulation-based  training. 

Task  Load  Index  Scores 

This  dependent  variable  sought  to  test  whether  computer-based  tools  reduce  the 
workload  of  plaiming  for  simulation-based  training  to  make  the  training  planner’s  ability 
to  plan  more  effectively.  The  TLX  scores  for  conducting  the  training  plaiming  (see 
Figure  6)  were  measured  to  ascertain  whether  the  difficulty  of  each  set  of  tasks  was 
reduced  using  the  manual  method  or  the  computer-based  planning  tool. 
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Figure  6.  Task  Load  Index  Scores  by  Factor 
Figure  6  presents  the  factors  involved  in  the  TLX  procedure.  First,  the  overall 
effects  and  statistics  of  the  total  TLX  score  will  be  examined.  Using  the  Mann- Whitney 
statistics  (Table  15),  the  TLX  scores  were  assessed  for  significance  as  relating  to 
efficiency.  The  median  TLX  score  for  the  manual  group  was  184.08  while  the  computer- 
based  group  had  a  median  score  of  1 66.67  (lower  is  better).  For  the  mean  and  standard 
deviation,  the  results  were  (mean„a„u3i  =  183.54,  stdev„,anuai  =  40.00)  and  (rnearia^omated  = 
166.47,  stdeVautomated  =40.43).  Using  the  Mann- Whitney  nonparametric  test,  the  first  part 
of  hypothesis  point  #3  is  not  confirmed  at  a  level  of  5%.  The  computer-based  planning 


TABLE  15 


TLX  SCORE  STATISTICS 


Variable 

Mediana„,„  -  Median^,^ 

P-value 

TLXScore 

-17.92 

0.065 

Mental  Demand 

-12.75 

0.012* 

Effort 

-11.25 

0.027* 

Physical 

Demand 

-1.0 

>0.05 

Temporal 

Demand 

1.5 

0.043** 

Frustration 

-4.34 

>0.05 

Performance 

10.5 

>0.05 

*  Automated  is  significantly  better  at  5%  *♦  Manual  is  significantly  better  at  5% 


tool  required  the  same  workload  and  was  thus  not  significantly  different  than  the  manual 
planning  method  in  efficiency. 

For  overall  efficiency  testing,  the  sum  of  the  six  factor  scores  were  used.  Table 
15  shows  the  results  of  the  statistical  tests  on  the  hypothesis  for  improving  the  efficiency 
of  simulation-based  training  plarming.  The  Mann- Whitney  statistical  testing  does  not 
verify  this  point  in  that  the  overall  difference  between  the  manual  and  computer-based 


groups’  perception  of  the  workloads  was  not  reflected  in  their  respective  planning 
processes. 

The  bipolar  factor  results  in  Figure  6  suggest  that  certain  factors  may  be  more 
important  than  others.  Three  TLX  bipolar  factors  showed  significant  differences  between 
the  manual  and  computer-based  groups.  For  Mental  Demand,  the  level  of  significance 
was  1 .2%  showing  that  the  computer-based  group  perceived  a  workload  factor  of  mental 
demand  99%  of  the  time  to  be  easier  than  that  experienced  by  the  manual  group.  Effort 
was  significant  at  a  level  of  2.7%  for  the  teams  using  the  computer-based  tool  versus  the 
manual  method.  With  the  introduction  of  a  new  tool,  the  perceived  temporal  demand  for 
the  automated  group  was  viewed  as  significantly  higher  for  the  automated  planning  tool 
users.  These  results  are  important  in  that  they  suggest  that  the  computer-based  planning 
reduces  workload  and  eventually  increases  effectiveness  as  defined  by  Rouse  (1991)  in 
terms  of  making  the  difficult  tasks  less  demanding.  The  temporal  demand  may  be  higher 
for  the  automated  teams  due  to  the  pressures  involved  in  trying  to  use  the  automated 
planning  tool  and  still  expect  to  finish  the  exercise  in  a  reasonable  period  of  time,  perhaps 
in  less  time  that  the  manual  teams. 

As  part  of  the  TLX  process  to  calculate  the  scores,  all  subjects  were  asked  to 
conduct  pairwise  weightings  of  the  six  dimensions  used  on  the  TLX  worksheet.  The 
significant  results,  of  mental  demand  and  effort  also  emanated  from  two  of  the  three 
highest  weighted  factors.  From  Figure  7,  the  significance  of  mental  demand  (25%)  and 
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effort  (21%)  having  predominant  weights  suggested  that  subjects  considered  these  factors 
more  demanding  in  the  experiment. 


Mental  Demand  physical 
25%  Demand 


Frustration 

13% 


Effort 

21% 


Temporal 

Demand 

17% 


Performance 

22% 


Figure  7.  Weights  for  TLX  Scoring 

The  mental  demand  and  effort  workload  dimensions  were  significantly  higher  for 
the  manual  teams  than  for  the  automated  teams.  However,  temporal  demand,  weighted 
at  17%,  was  perceived  by  the  automated  teams  as  being  higher  than  that  experienced  by 
the  manual  teams  (Table  1 5).  The  reason  for  such  a  result  may  be  that  the  automated^ 
teams  felt  pressured  by  time,  since  they  were  working  with  a  new  planning  tool. 

However,  the  effort  and  mental  demand  workloads  were  perceived  as  less  by  the 
automated  teams.  This  could  have  resulted  from  the  computer-based  tool’s  ability  to 
relieve  the  experimental  teams  from  the  tedious  and  mentally  demanding  tasks  of  training 
plan  development.  The  TLX  experimental  results  show  some  potential  of  the  ability  of  a 
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computer-based  planning  tool  to  increase  the  workload  efficiency  of  the  subjects.  An 
additional  measure  for  assessing  efficiency  is  the  time  required  for  conducting  the  tasks 
of  the  training  planning  experiment.  The  overall  times  were  compared  and  analyzed  to 
find  the  efficiency  differences  between  the  computer-based  and  manual  groups. 

Performance  Times 

The  task  performance  times  for  conducting  the  training  planning  (see  Table  16) 
were  measured  to  evaluate  the  time  required  throughout  the  experiment.  The  entire 
experiment  was  divided  into  four  separate  sessions.  The  overall  performance  time 
required  for  the  entire  exercise  was  used  as  the  dependent  variable  for  both  groups. 

The  Mann- Whitney  statistical  testing  does  not  support  the  premise  that  the 
automated  groups  had  lower  times  than  those  recorded  for  the  manual  planning  group 
(see  Table  16).  The  median  time  for  the  manual  group  was  250  minutes  while  the 
computer-based  group  had  a  median  time  of  224  minutes.  The  mean  and  standard 
deviations  for  time  performance  of  the  two  groups  are  (mean^anuai  272.2,  stdeVj„anuai  = 
80.5)  and  (meanautomated  =  238.9,  stdev^utomated  =57.2). 


TABLE  16 

PERFORMANCE  TIME  STATISTICS 


Median„,„ 

Median^nto 

P-value 

250 

224 

0.078 
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Applying  the  Mann- Whitney  nonparametric  statistical  test,  at  a  significance  level 
of  5%,  the  hypothesis  is  not  confirmed.  There  is  no  significant  difference  which  shows 
that  the  computer-based  planning  tool  required  less  time  and  was  thus  more  efficient  than 
the  manual  planning  method.  The  overall  time  for  training  planning  was  not  significantly 
different  for  the  two  groups.  Considering  that  the  computer-based  teams  were  initially 
trying  to  learn  how  to  use  the  computer-based  tool,  these  results  are  promising,  in  that  the 
automated  planning  teams  were  able  to  maintain  the  same  speed  as  the  manual  teams  in 
conducting  the  tasks. 

At  the  level  of  significance  of  5%,  the  third  point  of  the  hypothesis  was  not 
proven.  The  computer-based  planning  tool  failed  to  show  significant  improvement  in 
efficiency  of  training  plan  development  for  simulation-based  training.  A  summary  of  the 
dependent  measures  for  the  hypothesis  that  computer-based  planning  tools  enhance  the 
effectiveness  and  efficiency  of  simulation-based  training  is  provided  in  Table  17. 
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TABLE  17 


SUMMARY  OF  SIGNIFICANCE  FOR  DEPENDENT  VARIABLES 


Variable 

Mediauauto  -  Median„,an 

P-value* 

Free  Recall  ARC 
value  PostTest 

.22 

0.0389** 

Quality  Rating 

-1.5 

0.0002** 

Sim.MOE#3 

3 

0.096 

Performance  Time 

-23 

0.078 

TLX  Score 

-17.92 

0.065 

♦Level  of  Significance  is  5%;  **  Significant  at  p  <  0.05 
In  addition  to  the  statistical  results  for  proving  the  main  hj^othesis,  the  study 

gathered  positive  feedback  regarding  the  use  of  computer-based  planning  tools. 

User  Feedback 

As  for  the  computer-based  planning  tool,  feedback  was  very  positive  with 
exclamations  on  how  essential  a  training  planning  tool  is  (see  Appendix  S).  Some  of  the 
novice  planners  stated  that  they  understood  the  training  planning  process  better  and 
appreciated  the  opportunity  to  learn  about  training  and  planning  with  simulations  (see 
Table  18).  Many  subjects  expressed  the  desire  to  have  more  time  with  the  planning  tool 
and  the  simulation. 
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TABLE  18 


POST-EXERCISE  FEEDBACK 
What  did  users  like  about  the  TREDS planning  tool? 


Training 

Planning 

Mental  Models 

Computer-based 

“It  was  good  to 
be  able  to  see 
what  areas  you 
need  to  train  for 
each  task  and 
howto 

incorporate  those 
tasks  into 
training.” 

“Determining 
weaknesses  in  my 
planning  process, 
but  also,  able  to  see 
how  the  plaiming 
process  can  be 
effective.” 

“The  computer  gave 
you  a  picture  to  work 
with.  You  could 
easily  try  different 
approaches  quickly.” 

“It  was  much 

more  user 
friendly  than 
trying  to  do  it 
‘stubby  pencil’." 

Overall  “It's  a  good  training  tool;  best  I've  seen  in  19 

comment:  years!” 

Overall,  the  majority  of  the  subjects  provided  positive  and  useful  feedback  to 
improve  the  training  planning  tool.  Prior  to  the  experiment,  most  of  the  subjects  had 
never  seen  or  used  a  computer-based  training  planning  tool.  The  initial  demographics  of 
each  subject  suggested  that  there  were  adequate  levels  of  computer  experience  and 
Microsoft  Windows©  knowledge  than  originally  anticipated  by  the  experimenter. 
However,  the  feedback  received  after  the  testing  was  very  informative  in  understanding 
the  expectations  and  needs  of  the  subjects. 
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User  feedback  accentuated  the  need  for  automated  planning  tools  for  simulation- 
based  team  training.  Also,  those  subjects  who  had  little  experience  with  the  Janus 
simulation  provided  positive  feedback  for  the  maimer  in  which  the  automated  planning 
tool  helped  them  understand  and  use  the  simulation  better.  Many  individuals,  including 
those  on  the  manual  planning  teams,  were  anxious  to  return  for  more  simulation-based 
team  training. 


Summary 

The  results  of  the  experiment  encourage  further  research  and  development  of 
computer-based  planning  tools  for  simulation-based  training.  The  methods  used  to  obtain 
viable  results  were  instrumental  in  the  experiment’s  success.  By  using  appropriate  and 
measurable  criteria  for  the  treatment  effects,  the  study  captured  iimovative  and  reliable 
methods  to  justify  computer-based  planning.  Each  measurement  yielded  information  on 
the  potential  need  for  future  research  of  computer-based  planning  tools. 

The  ability  of  the  TREDS  computer-based  planning  tool  to  improve  the  user’s 
mental  model  of  the  training  planning  process  shows  knowledge  improvements  for 
novice  planners.  The  results  of  the  study  did  not  show  any  significant  mental  model 
increase  for  the  manual  planning  teams.  Participation  in  the  experiment  did  not  seem  to 
increase  their  imderstanding  of  the  training  planning  process. 

For  this  experiment,  a  quality  rating  form  was  used  to  gather  feedback  on  the 
overall  quality  of  the  training  planning  method.  The  effectiveness  of  the  planning  tool  is 
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essential  to  convince  users  to  reuse  the  tool.  The  quality  rating  score  was  significant  for 
showing  the  effectiveness  of  the  overall  training  planning  process.  The  simulation 
measures  of  effectiveness  results  supported  the  premise  that  the  teams  that  used 
automated  planning  tool  developed  more  effective  training  plans. 

Finally,  the  measures  of  efficiency  were  evaluated  to  determine  the  utility  of  an 
automated  planning  tool  to  enhance  simulation-based  training.  The  TLX  scores  and 
performance  time  results  showed  significant  improvement  in  the  efficiency  of  computer- 
based  teams  in  planning  the  essential  tasks  for  simulation-based  training.  All  three  points 
of  the  hypothesis  were  not  confirmed  at  the  5%  level  of  significance,  indicating  that  the 
use  of  computer-based  planning  tools  does  not  enhance  the  effectiveness  and  efficiency 
of  simulation-based  training.  This  statement  is  somewhat  contradicted  by  the  positive 
user  feedback  gleaned  after  the  experiment.  The  next  chapter  will  conclude  the  study  and 
recommend  future  courses  of  action  to  capitalize  on  the  contributions  of  this  research 
which  warrant  further  examination  of  computer-based  planning  for  simulation-based 
training. 
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CHAPTERS 


DISCUSSION  AND  RECOMMENDATION 

The  original  contribution  of  the  research  conducted  in  this  study  is  the  partial 

validation  of  benefits  derived  from  the  use  of  computer-based  planning  tools  to  enhance 
the  effectiveness  and  efficiency  of  simulation-based  team  training.  Specifically,  the 
study: 

1 .  provides  a  methodology  for  investigating  the  use  of  computer-based  planning 
tools  to  enhance  the  effectiveness  and  efficiency  of  simulation-based  training, 

2.  identifies  the  linkage  between  training  task  requirements  and  simulation-based 
training  characteristics  by  building  a  mental  model  of  the  training  plan 
development  process,  and 

3.  assists  future  research  efforts  by  providing  potential  measures  of  effectiveness 
and  efficiency  for  evaluating  technological  advancements. 


Discussion 

The  major  question  related  to  this  study  has  been:  What  contributions  do 
computer-based  planning  tools  offer  to  simulation-based  team  training?  This  question  is 
divided  into  three  points: 


1.  Mental  Model  Measurement:  Will  a  computer-based  planning  tool  enhance 
the  user’s  mental  model  of  the  training  plan  development  process? 

2.  Measures  of  Training  Effectiveness:  Are  more  effective  training  plans  created 
and  better  results  achieved  with  computer-based  planning  tools? 

118 


3.  Measures  of  Training  Efficiency:  Is  simulation-based  training  planning  more 
efficiently  accomplished  using  computer-based  tools? 

Mental  Model  Measurement 

Hypothesis  Point  #1:  A  computer-based  planning  tool  will  enhance  the  mental 
model  of  the  training  plan  development  process.  Based  on  the  results  this  hypothesis  was 
supported.  The  use  of  cognitive-based  training  exercise  development  systems  for 
planning  simulation-based  training  has  the  potential  to  improve  the  imderstanding  of 
generating  training  development  plans.  Other  applications  of  computer-based  planning 
tools  for  simulation-based  team  training  could  assist  civilian  and  military  agencies  in 
planning  and  conducting  disaster  preparedness  training,  airline  pilot  exercises,  and 
nuclear  accident  scenario  drills. 

The  study  has  collected  potential  justification  for  the  enhancement  of  simulation- 
based  training  planning  by  using  a  computer-based  planning  tool.  The  method  to  prove 
such  a  theory  applied  human  cognition  factors  in  the  development  of  mental  models. 
While  the  conceptual  model  of  a  system  or  process  is  likely  to  evolve,  the  users  need  to 
understand  the  structure  of  the  process  in  order  to  foster  an  accurate  mental  model.  This 
study  has  provided  a  means  to  determine  the  user’s  mental  model  for  training  planning 
and  measure  how  effective  that  mental  model  has  developed  using  free  recall  ARC 
values. 
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Measures  of  Training  Effectiveness 


Hypothesis  Point  #2:  More  effective  training  plans  are  created  and  results 
achieved  with  computer-based  planning  tools.  This  point  was  partially  substantiated  with 
the  data  from  the  quality  ratings.  The  results  of  the  simulation  MOE  do  not  show 
significant  difference  between  the  two  groups  at  an  alpha  of  5%.  The  computer-based 
planning  teams  considered  the  utility  of  their  tool  to  be  of  higher  quality  and  thus  more 
effective  than  the  manual  tool  planning  teams  considered  theirs.  The  capability  of  the 
computer-based  planning  tool  to  enable  users  to  link  training  tasks  objectives  with  the 
simulation-based  training  was  achieved. 

With  respect  to  the  simidation  exercise,  the  computer-based  planner’s  measures  of 
effectiveness  for  the  survivability  of  tnendly  systems  or  destruction  of  enemy  systems 
was  not  significantly  higher  than  those  for  the  manual  planners.  Based  on  significant 
results  with  only  the  quality  ratings  results,  these  results  cannot  affirm  that  a  computer- 
based  planning  tool  creates  more  effective  training  plans  for  improved  simulation  results. 

Measures  of  Efficiency 

Hypothesis  point  #3:  Simulation-based  training  planning  is  more  efficiently 
accomplished  using  computer-based  tools.  First  of  all,  the  Task  Load  Index  (TLX) 
results  show  that  the  computer-based  planning  tool  did  not  decrease  or  increase  the 
perceived  overall  workload  for  the  treatment  group.  Specifically,  mental  demand  and 
effort  were  the  two  factors  that  the  computer-based  planning  group  reported  as 
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significantly  easier  than  that  experienced  by  the  manual  planning  group.  However,  the 
manual  planning  teams  perceived  that  the  temporal  demands  of  their  methods  were  less 
than  those  reported  by  the  automated  tool.  This  could  have  occurred  due  to  the  time  that 
the  automated  users  needed  in  order  to  become  familiar  with  the  computer-based 
planning  tool. 

For  overall  performance  time,  there  were  no  significant  differences  between  the 
two  groups  at  a  level  of  significance  of  5%.  The  times  between  the  two  locations  showed 
a  large  increase  in  time  because  of  the  availability  of  full-time  simulation  center 
instructors  at  the  second  location  (Fort  Hood).  Finally,  one  of  the  key  objectives  for 
computer-based  planning  tools,  as  also  for  the  TREDS  program,  is  to  save  time  for  team 
planners.  Once  the  users  improve  their  use  of  the  TREDS  program  in  its  Microsoft 
Windows©  environment,  the  training  planning  time  for  automated  planners  could 
decrease  further  with  an  increased  understanding  of  the  peripheral  TREDS  programs  and 
its  operating  system. 

Investment  for  simulations  is  costly  in  terms  of  time  and  manpower.  Thus,  the 
need  to  ensure  their  full  utilization  is  very  critical  to  ensure  effective  and  efficient  user 
implementation  and  training  benefits.  Delivering  effective  training  at  a  significantly 
reduced  cost  is  becoming  a  priority  for  planners  and  trainees.  The  potential  of  TREDS 
for  meeting  the  needs  of  those  involved  in  the  complexities  of  the  simulation-based 
training  planning  process  has  been  indicated,  but  not  significantly  proven  through  the 
analysis  of  the  hypothesis’  components.  The  results  did  confirm  that  automated  planning 
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tools  significantly  aid  the  user’s  understanding  of  the  conceptual  training  plan 
development  processthan  the  manual  planning  methods  do.  Based  on  the  potential  of  the 
study  to  meet  training  planners’  needs,  there  are  areas  in  TREDS  which  need  to  be 
addressed  for  future  development  and  validation  of  automated  training  planning  tools  for 
simulation-based  training. 


Recommendations 

Academic,  industry  and  government  trainers  must  exploit  computer-based 
planning  and  evaluation  tools  to  improve  the  effectiveness  and  efficiency  of  simulation- 
based  training.  In  this  realm,  this  study  has  started  the  process  and  tested  a  means  for 
measuring  technology’s  impact  on  the  user.  Findings  of  the  present  study  may  not  be 
representative  of  other  groups  consisting  of  different  populations  since  the  military 
population  has  unique  objectives  in  conducting  simulation-based  training.  The  study 
could  yield  more  significant  results  in  a  different  environment. 

In  essence,  the  effects  of  technology  are  being  measured  using  a  human-centered 
approach,  versus  the  usual  manner  of  focusing  on  the  machine.  Schneiderman  (1992) 
challenges  that  the  term  ‘user-fnendliness’  should  be  replaced  by  measurable  human- 
factors  quality  criteria  such  as: 

1.  time  to  learn, 

2.  speed  of  performance, 

3.  rate  of  errors  by  users, 
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4.  retention  over  time,  and 


5.  subjective  satisfaction,  (p.  82) 

This  study  has  applied  only  two  of  the  above-mentioned  variables — speed  of 
training  plan  development  and  subjective  user  satisfaction.  In  order  to  substantiate  that 
the  study’s  hypothesis  can  be  proven,  further  research  should  test  computer-based 
planning  tools  applying  the  three  remaining  criteria —  (1),  (3)  and  (4).  “Time  to  learn” 
and  “retention  over  time”  could  be  measured  using  a  extended  study  testing  period.  With 
a  similar  experimental  design  set  up  to  measure  these  criteria,  the  computer-based 
planning  tool  could  show  significant  increases  in  effectiveness  and  efficiency. 

This  study  examined  the  capability  of  a  computer-based  planning  tool  to  assist 
users  in  building  a  mental  model  of  the  training  planning  system.  According  to  Eberts 
(1994)  additional  feedback  capabilities  for  a  final  design  of  a  system  should  follow  these 
guidelines: 

1 .  users  need  feedback  to  develop  accurate  mental  models, 

2.  the  system  should  provide  feedback  which  corresponds  to  goals  and  plans  of 
the  user,  and 

3.  the  system  should  assess  the  goals  and  plans  while  providing  feedback  on 
assessment  (p.  525). 

Once  the  TREDS  planning  tool  is  completed  and  tested  in  its  initial  domain,  these 
points  should  be  incorporated. 
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Other  areas  of  simulation-based  training  should  be  considered  for  implementation 
of  a  computer-based  planning  tool.  Following  a  proof-of-principle  demonstration  in 
August  1995  with  Orange  County,  Florida,  the  Plowshares  program  will  continue  to 
“apply  military  constructive  simulation  technology  to  training  and  analysis  for  emergency 
management”  (Petty  &  West,  1995).  Included  with  the  Plowshares  program  is  a 
derivative  of  TREDS  which  is  the  System  for  Training  Emergency  Personnel  (STEP) 
(McGinnis,  1994).  STEP  is  a  computer-based  planning  tool  which  will  assist  in  the 
overall  planning  and  assessment  of  the  Plowshares  simulation-based  emergency  team 
training  exercises  (McGinnis,  1994). 

Other  application  areas  for  automated  planning  in  simulation-based  team  training 
include  the  nuclear  power  plant  industry,  aviation  simulator  programs,  space  exploration, 
sports  and  business  strategic  executive  simulations.  Sports  training  could  also  apply  a 
computer-based  training  planning  tool  to  assist  in  the  analysis  of  team  or  collective  tasks. 
A  team  has  a  mission  that  it  often  simulates  during  practice  on  how  to  score  points  in  a 
football  game  or  goals  in  a  soccer  match.  These  sports  already  conduct  simulated 
scouting  exercises  each  week  when  they  try  to  guess  how  their  opponent  will  play  in  the 
next  game.  A  computer-based  system  would  serve  to  save  time  and  efforts  for  future 
replays  against  similar  teams  annually,  while  tracking  the  progress  of  the  team  to  perform 
its  assigned  tasks. 

One  benefit  of  having  the  computer-based  training  planning  tool  replicate  expert 
models  of  the  training  planning  process  is  suggested  by  Totterdell,  Norman,  and  Browne, 
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(1987)  in  that  the  best  technology  may  be  the  kind  which  adjusts  to  the  operator  instead 
of  compelling  users  to  tailor  to  the  technology.  As  engineers  and  computer  programmers 
create  new  systems,  they  should  follow  the  user  design  guidance  offered  in  the  human- 
computer  interaction  literature.  Validating  this  point,  Helander  (1988)  asserts  that  the 
cognitive  psychologists  must  develop  guidelines  for  the  designers  of  computer  systems  in 
order  to  match  human  capabilities  with  complex  system  requirements. 

The  future  direction  in  computer-based  technological  implementations  is 
continuously  exploited  in  our  daily  lives.  The  scientific  application  of  knowledge-based 
systems  to  help  users  navigate  through  software  may  be  a  starting  point  for  bridging  the 
gap  between  system  designers  (represented  by  computer  programmers  and  engineers)  and 
users  (represented  by  the  psychologists).  As  computer-based  planning  tools  surface,  the 
same  theories  for  mental  model  replication  and  evaluation  used  in  this  study  should  be 
applied.  Understanding  the  cognitive  mental  models  of  the  users  must  still  be  considered 
before  this  technology  evolves  further. 

Once  a  user  becomes  very  familiar  with  the  planning  process,  their  mental  model 
is  mature,  dispensing  the  need  for  an  elaborate  plarming  tool.  According  to  Eliot  (1994) 
intelligent  agents  are  “software  that  acts  on  the  behalf  of  a  human  master  to  achieve  a 
desired  goal”  (p.  9).  At  the  point  of  complete  familiarity  with  a  process,  the  computer- 
based  planning  tool  could  be  transformed  into  an  intelligent  agent  which  could  replicate 
the  user’s  mental  model.  The  intelligent  agent  would  perform  training  planning 
development  as  the  user  and  eventually  the  agent  could  become  a  super-agent  by 
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collating  and  synthesizing  all  of  the  training  planning  experts  into  an  super  expert  training 
planner  agent.  Rouse  and  Howard  (1993)  support  this  conclusion  by  predicting  that; 

Trends  in  computer  and  communications  technologies  promise  to  make 
these  types  of  support  system  increasingly  powerful  and  widely  available. 

It  is  easy  to  envision  integrated  planning  systems,  utilizing  multiple, 
enterprise-tailored  advisors.  In  addition,  multimedia,  simulation,  and 
groupware  technologies  can  enable  highly  interactive,  distributed  planning 
(p.  53). 

Summary 

As  Brynjolfsson  (1993)  aptly  points  out,  the  role  of  information  technology’s  (IT) 
contribution  to  productivity  has  yet  to  be  determined  due  to  inadequate  consensus  and 
proof  of  reliable  IT  measurements.  Brynjolfsson  (1993)  states  that  the  main  issue  is  not 
how  fast  or  how  much  more  we  will  be  able  to  do  with  technology,  but  that  IT  will  enable 
us  “to  do  entirely  new  things  in  new  ways”  (p.  76).  By  choosing  the  appropriate 
measuring  tool  for  unconventional  methods,  leaders  and  managers  will  be  able  to  define 
the  benefits  of  the  technology. 

Today,  there  are  many  statements  regarding  the  production  enhancements  effected 
by  automation.  However,  the  data  to  support  such  claims  are  often  unfounded.  Through 
the  use  and  testing  of  a  computer-based  planning  tool,  the  results  reported  in  this  paper 
serve  to  strengthen  the  premise  that  automation  does  enhance  productivity. 

The  purpose  of  this  paper  was  to  utilize  behavioral  theory  to  support  the  rationale 
for  the  use  of  computer-based  planning  tools  for  simulation-based  training.  Data 
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collection  which  applied  the  computer-based  planning  tool  in  an  experimental  setting 
produced  results  to  show  that  this  new  tool  produces  better  results,  both  in  effectiveness 
and  efficiency.  These  findings  are  important  in  that  they  point  to  ways  to  show 
technology’s  impact  in  a  quantitative  manner  for  cognitive  tasks.  The  automation  offered 
by  technology  can  save  people  time  and  effort. 

While  mankind  proliferates  automation  applications  from  the  workplace  to  the 
home,  many  people  seem  to  spend  more  time  in  front  of  a  computer  screen  than  enjoying 
life.  The  novelist  Dean  Koontz  (1994)  implies  that  the  effects  of  automation  is  not 
always  a  positive  consequence: 


New  technology — like  the  computer — ^freed  men  and  women  from  all 
kinds  of  drudgery,  saved  them  vast  amounts  of  time.  And  yet ...  and  yet 
the  time  saved  did  not  seem  to  mean  additional  leisure  or  greater 
opportunities  for  meditation  and  reflection.  Instead,  with  each  new  wave 
of  technology,  the  pace  of  life  increased;  there  was  more  to  do,  more 
choices  to  make,  more  things  to  experience,  and  people  eagerly  seized 
upon  those  experiences  and  filled  the  hours  that  had  only  moments  ago 
become  empty  (p.  273). 

With  the  extra  time  and  less  effort  afforded  by  computer-based  planning  tools,  the 
intent  is  not  to  do  more,  but  to  fill  the  time  and  add  to  life  the  little  things  that  subtly 
demand  attention  and  often  provide  great  reward.  It  is  with  this  hope  and  purpose  that  the 
reader  finds  utility  in  applying  automation  to  enhance  the  effectiveness  and  efficiency  of 
processes  such  as  planning. 
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Appendix  A 


List  of  Abbreviations  and  Acronyms 
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ADD:  Automated  exercise  Distribution  and  Display 

APEP:  Automated  exercise  Preparation,  Evaluation  and  Preview 

ARC;  Adjusted  Ratio  of  Clustering 

ARE:  Asymptotic  Relative  Efficiency 

ASG:  Automatic  Scenario  Generator 

ATOM:  Anti-Air  Warfare  Teamwork  Observation  Measure 

BOS:  Battlefield  Operating  System 

DIS:  Distributed  Interactive  Simulation 

GOMS:  Goals,  Operators,  Method,  Selection  rules 

ISD:  Instructional  Systems  Design 

IT:  Information  Technology 

METL:  Mission  Essential  Task  List 

MOE:  Measure  of  Effectiveness 

NAWCTSD;  Naval  Air  Warfare  Center  Training  Systems  Division 

NPP:  Nuclear  Power  Plant 

PMOE:  Programmatic  Measures  of  Effectiveness 

SATS:  Standard  Army  Training  System 

SME:  Subject  Matter  Expert 

TADMUS:  Tactical  Decision  Making  Under  Stress 

TEFL:  Training  Exercise  Force  Laydown 

TREDS:  TRaining  Exercise  Development  System 

TVS:  Training  Value  System 
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Demographics  Questionnaire 
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DEMOGRAPHICS  QUESTIONNAIRE 

Date: 


All  information  will  be  kept  in  strict  confidence.  Only  aggregate  data  will  be  reported. 
Please  provide  the  following  information: 

A.  GeneraL 

1.  Name  (optional): _ 2.  Unit: _ 

3.  Duty  Position: _  4.  Rank: _ 

5.  Age: _  6.  Time  in  Service: _ 

B.  Computer-related  skills. 

1 .  Rate  your  experience  with  personal  computers  (circle  the  appropriate  response): 


2.  If  you  related  your  experience  with  personal  computers  in  terms  of  years,  where  would 
you  place  yourself  on  the  chart  below?  (circle  the  appropriate  response): 


1  2 

3  4  5  6 

7  8 

9  10 

lone 

more  thin  6  months 

ovlr  2  years 

over  i  years 

3.  Approximately  how  often  do  you  use  a  personal  conqjuter  each  month?  (circle  the 
appropriate  response): 

1  2  3  4  5  6  7  8  9  10 

lever  once  L  week  2-3  times/week  daily 
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4.  Do  you  own  a  personal  computer? 


Yes 


No 


5.  Have  you  used  Windows  before?  Yes  |  |  No  |  | 

6.  Where  do  you  consider  your  academic  preferences  to  be?  (circle  a  number 
the  closest  to  your  interests) 


1 _ 2 _ 3 _ 4 


English/  Science  Math  Computers 
Humanities 


5 


Other 


C.  Training  and  Simulation 

1.  If  you  related  your  unit  training  planning  experience  in  terms  of  years,  where 
you  place  yourself  on  the  chart  below?  (circle  the  appropriate 


12  3  4 


none  more  than  6  months  over  2  years  over  5  years 


8.  In  the  last  year,  how  many  hours  have  you  spent  training  in  a  simulation  or 
device? 


1  2  3  4 

I  I  ^  I 

None  at  least  10  at  least  25  more  than  50 


9.  Have  you  used  any  automated  tools  in  planning  for  training  Yes  Q  No 
If  yes,  which  training  tools  have  you  used? 


10.  If  applicable,  which  simulation  or  simulator  devices  have  you  trained 
SIMNET  Q  Janus  Q  BBS  Q  Other: _ 


This  concludes  the  Demographic  Questionnaire.  Thank  you  for  your  assistance. 
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Appendix  C 


Post-Exercise  Questionnaire 
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POST  EXERCISE  QUESTIONNAIRE 


Team  # _  Date _ 

Name  (optional): _  Rank _ 

All  information  will  be  kept  in  strict  confidence.  Only  aggregate  information  will  be 
reported. 

Please  answer  the  following  questions  by  circling  your  response: 

1 .  How  well  did  this  exercise  increase  your  knowledge  of  the  training  process? 


average 


extensive 


2.  How  similar  was  this  training  exercise  to  your  current  planning  process  for  unit 
training? 


somewhat  close 


similar 


very  similar 


3.  How  would  your  peers  in  other  units  benefit  from  such  a  training  system? 


average 


extensive 


4.  This  planning  process  for  unit  training  would  be  good  for  training  the  following 
(mark  all  of  the  entries  you  feel  are  appropriate): 


Individual 


Q  Crew  Q  Platoon  Q  Company  Q 


Battalion  Staff 


Other: 
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5.  What  areas  do  you  consider  need  improvement  in  the  training  planning 


process? 


6.  What  did  you  like  most  about  this  exercise? 


7.  What  would  you  change  to  make  this  a  better  exercise? 


8.  Any  other  comments  you  have  to  contribute  would  be  greatly  appreciated. 


Thank  you  for  your  time  and  assistance  in  this  exercise. 
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Practical  Exercise  for  Experiment 
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SATS-TREDS  Janus  Practical  Exercise 
Automated  Planning  for  Simulation-based  Training 
PMCATT- TREDS-Xl  Date:  8  May  95 

I.  General  information: 

A.  This  practical  exercises  seeks  to  integrate  training  needs  and  objectives 
with  simulation-based  training.  At  home  station,  leaders  evaluate  the  effectiveness  of 
exercise  training  plans  on  the  battlefield,  in  NTC  livefire  exercises,  during  SIMNET 
scenarios,  or  in  Janus  combat  simulation  exercises. 

B.  Time  is  always  a  limited  resource  for  planning  training.  Building  a 
training  exercise  plan  requires  an  organized  structure  of  specific  aspects  in  each  planning 
phase.  Any  comments  you  have  for  improving  this  process  will  be  helpful  for  future 
development  for  all  Army  training  systems. 

n.  Situation: 

A.  Your  company,  the  world-renown  Mike  Company  of  TF  3-67  AR,  is 
planning  training  for  the  fourth  quarter  of  FY  1995,  July  to  September.  Key  simulation 
training  objectives  are  outlined  in  the  battalion’s  quarterly  training  guidance  (see 
enclosure  1).  The  Task  Force’s  and  company’s  training  highlights  are  in  enclosure  2. 


FOR  TRAINING  PURPOSES  ONLY 
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Based  on  Mike  Company ’s  November  1994  Co/Tm  Level  ARTEP  results  (see 
end  4),  the  BN  CDR  wants  each  company  to  improve  proficiency  in  as  many  tasks  as 
possible  using  the  cost-effective  training  offered  by  the  simulation  centers. 

B.  Each  company  commander  should  work  closely  with  the  S-3  staff  on 
all  training  planning.  The  BN  CDR  has  directed  the  S-3  to  assist  with  all  planning.  Use 
the  November  ARTEP  results  to  orient  on  specific  tasks.  For  the  September  NTC 
rotation,  Mike  Company  will  be  the  lead  force  in  the  defense  missions  for  the  Task  Force. 

III.  Requirements: 


Training  Plan  Development 

Ql:  Does  automated  planning  improve  efficiency 
and  effectiveness  of  training  plan  development? 


Session  #1 


•Goals 

•Objectives 

•Training  Needs 

•Resources 

•Constraints 

•Performance 

Measures 


Manual 


Automated 
Planning 


Session  #2 


'Outputs.  P 


•Training  Plan 
0OPORD 
0Overlay 
0Exec  Matrix 
•Simulation  Input 


Phase  I:  Training  Plan  Development 
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Phase  II:  Simulation-based  Training  &  Assessment 
A.  Phase  I:  The  Training  Plan  Development  Phase  consists  of  the 
following  two  sessions: 

1 .  Review  the  training  calendar,  ARTEP  results,  current  unit 
METL  (end  3),  MTPs,  FMs  and  commander’s  training  guidance  to  prioritize  the  tasks  for 
training  in  a  typical  Janus  scenario  (duration  of  training  period  not  to  exceed  three  hours). 

2.  Develop  paragraphs  2  and  3  of  the  Operations  Order  (or  the 
OPORD  Matrix),  an  Operations  Overlay,  coordinated  through  an  execution/OPORD 
Matrix  for  planning  a  Janus  simulation  exercise. 


FOR  TRAINING  PURPOSES  ONLY 
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B.  Phase  11;  The  Simulation  Training  Phase  consists  of  two  sessions: 

1 .  Establish  or  review  the  measures  for  success  on  specified 
standards  for  the  tasks  in  the  scenario  you  have  developed.  Observe  the  simulation  and 
record  the  values  for  the  selected  measures  during  and/or  after  the  simulation’s  execution. 

2.  Using  the  results  of  the  simulation,  assess  the  new  proficiency 
ratings  for  only  those  tasks  in  the  scenario.  This  will  involve  rolling  up  the  results  of  the 
standards  into  subtasks,  then  into  tasks  ratings. 

IV.  The  Janus  Simulation.  The  Janus  combat  simulation  model  will  be  available 
for  the  battalion  to  train  on  during  selected  dates  in  the  fourth  quarter.  As  part  of  this 
exercise,  there  will  be  an  initial  Janus  “train-up”  session  for  participants  to  leam  Janus 
fimdamentals.  Janus  lends  itself  well  to  conduct  staff  and  leader  training.  After-action 
review  and  “what-if  ’  analyses  can  also  conducted  using  Janus.  Janus  simulation  output  is 
to  be  recorded  via  the  Janus  post-processing  file  system. 


GEORGE  F.  STONE  III 
MAJ,FA 

Ends 

1:  Bn  Tmg  Guidance 
2:  M  Co  Tmg  Highlights 
3:  MCoMETL 
4:  M  Co  ARTEP  Results 
5:  MOE  for  Simulations 


FOR  TRAINING  PURPOSES  ONLY 
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3d  Battalion,  67th  Armor 
2nd  Armored  Division 
Ft.  Hood,  TX 


6  June  1995 

Subject:  Battalion  Training  Guidance  for  4th  Quarter,  FY95  (Enclosure  1) 

1.  General:  This  guidance  is  for  all  personnel  in  3-67  AR.  It  will  be  modified  as 
needed  to  meet  unit  training  objectives  and  METL. 

2.  Specific: 


a)  Units  will  fully  utilize  all  division  simulation  assets  to  prepare  for 
quarterly  and  aimual  training  exercises.  The  Janus  simulation  will  be 
available  throughout  the  quarter  for  eidiancing  proficiency  on  leadership  and 
collective  tasks. 

b)  The  Janus  simulation  tools  enable  the  application  of  measures  of 
effectiveness  to  evaluate  Go/NoGo  standards  for  MTP  standards.  These 
standards  have  also  been  assigned  Task  Performance  Support  codes  for  their 
applicable  training  capability  in  Janus.  The  battalion’s  goal  for  increasing  the 
nmnber  of  Go’s  on  MTP  standards  through  simvilation-based  training  is  25%, 

c)  Manual  and  automated  planning  methods  will  be  essential  for  bringing 
training  simulation  to  all  soldiers  and  leaders.  All  leaders  should  strive  to 
improve  these  tools  through  feedback  to  the  simulation  system  managers. 

d)  Leaders  should  not  be  stifled  by  the  limitations  of  simulations.  Innovation 
and  initiative  should  be  prevalent  in  finding  ways  to  maximize  simulation 
utility.  To  maximize  the  benefits  of  training  simulations,  leaders  must 
become  aware  of  each  simulation’s  capabilities  and  limitations.  Reading 
simulation  reference  material  and  using  computer-based  training  programs 
will  assist  greatly. 
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HOOD 

LTC,AR 


Enclosure  2:  Mike  Company’s  4th  Quarter  (FY  95)  Training  Highlights 


July 

BN  Staff  Janus  CPX  (5-7) 
SIMNET  Platoon  Exercises  (12-15) 

August 

Task  Force  Leaders  Janus  CPX  (3-5) 
Task  Force  FTX  (13-16) 


NTC  Rotation  (2-29) 


Enclosures:  UnitMETL 


Mike  Company,  3-67  AR  Mission  Essential  Tasks  List 

References.  ARTEP  71-1-MTP,  3-67  AR  Battalion  METL,  BN  Training 
Guidance,  Mike  Company  Training  Guidance. 

General.  The  following  tasks  will  be  those  mission  essential  tasks  that  Mike 
Company  has  to  perform  in  order  to  fulfill  its  wartime  preparation  requirements.  All 
other  specified  and  implied  tasks  will  be  added  as  appropriate. 


BOS:  Maneuver 


17-2-0323 

Withdraw  imder  enemy  pressure 

Partially  trained  (P) 

17-2-0322 

Withdraw  not  under  enemy  pressure 

Trained  (T) 

17-2-1021 

Defend 

Partially  trained  (P) 

17-2-0306 

Support  by  Fire 

Untrained  (U) 

17-2-0202 

Perform  Reconnaissance 

Untrained  (U) 

17-2-0302 

Perform  Tactical  Road  March 

Untrained  (U) 

BOS:  Mobility  and  Survivability 


TaskKmnber 

17-2-0502 

Emplace  an  Obstacle 

Untrained  (U) 

BOS:  Intelligence 


Ta^  Humber 

17-2-0201 

Maintain  Operation  Security 

Partially  trained  (P) 

FOR  TRAINING  PURPOSES  ONLY 
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Enclosure  4:  Mike  Company’s  November  ARTEP  Results  with  Task 
Performance  Support  codes  for  the  Janus  combat  simxilation. 

(Document  is  not  enclosed  due  to  its  length). 


Disclaimer:  The  enclosed  ARTEP  results  are  fictitious  and  do  not  represent  any 

unit. 


145 


Appendix  E 


Cognitive  Analysis  Tool  Program  Rules 
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MAINGOAL  "Training  Exercise  Development  OLEsystem"; 

RULE  "Training  Exercise  Development  OLEsystem"  "This  model  portrays  the 
current  conceptual  layout  of  TREDS,  vs.  1.0,  dated  January  1995."; 

METHOD  INCOMPLETE  "Training  Exercise  Development"; 

STEP  "Events", 

STEP  "Tasks", 

STEP  "Scenarios"; 

END; 

END; 

RULE  "Events"; 

METHOD  INCOMPLETE  "The  Events  are  Scheduled  Occurrences  on  the  Unit’s 
Training  Calendar"; 

STEP  GROUP  "Record  METL", 

STEP  "Build  Calendars  &  Schedules"; 

END; 

END; 

RULE  "Tasks"; 

METHOD  INCOMPLETE  "Tasks"; 

STEP  "Assess  Training  Proficiency", 

STEP  "Prioritize  Tasks", 

STEP  "Allocate  Tasks  to  Events"; 
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END; 


END; 

RULE  "Scenarios"; 

METHOD  INCOMPLETE  "Scenarios"; 

STEP  "Select  Training  Conditions", 

STEP  "Select  Scenarios", 

STEP  "Edit  Scenarios", 

STEP  "Check  TADSS  Constraints", 

STEP  "Build  Input  Files  &  Exercise  Products", 

STEP  "Capture  Assessments"; 

END; 

END; 

RULE  "Record  METL"; 

METHOD  INCOMPLETE  "Record  METL"; 

STEP  "Review  Current  METL", 

STEP  "Update  METL"; 

END; 

END; 

RULE  "Build  Calendars  &  Schedules"; 

METHOD  INCOMPLETE  "Build  Calendars  &  Schedules"; 
STEP  "Import  Calendar  from  higher  Hqs", 
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STEP  "Review  Training  Guidance", 

STEP  "Link  training  activities  with  guidance", 
STEP  "Review  final  training  schedule"; 

END; 


END; 

RULE  "Assess  Training  Proficiency"; 

METHOD  INCOMPLETE  "Assess  Training  Proficiency"; 
STEP  "Review  current  proficiency  ratings", 

STEP  "Review  previous  training  evaluations", 

STEP  "Update  training  proficiency  status"; 

END; 


END; 

RULE  "Prioritize  Tasks"; 

METHOD  INCOMPLETE  "Prioritize  Tasks"; 


STEP  "Assign  commander's  guidance  to  tasks", 
STEP  "Rank  tasks  by  weights"; 

END; 

END; 

RULE  "Allocate  Tasks  to  Events"; 

METHOD  INCOMPLETE  "Allocate  Tasks  to  Events"; 
STEP  "Select  the  Training  Event", 
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STEP  "Confirm/revise  Current  Event  Schedule", 

STEP  "Notify  Persoimel  of  Schedule", 

STEP  "Identify  Tasks  that  event  can  train", 

STEP  "Select  Tasks  to  be  trained  in  event"; 

END; 

END; 

RULE  "Select  Training  Conditions"; 

METHOD  "Low  Level  Difficulty"; 

STEP  "Review  METT-T  Parameters", 

STEP  "Specify  Low  Level  Parameters"; 

IF  "Lack  of  Familiarity  with  Training  Device" 

TRY  METHOD  "Low  Level  DifTiculty"; 

IF  "Low  Training  Proficiency  on  Device" 

TRY  METHOD  "Low  Level  Difficulty"; 

END; 

METHOD  "Medium  Level  Difficulty"; 

STEP  "Review  METT-T  Parameters", 

STEP  "Specify  Medium  Level  Parameter  Values"; 

IF  "At  least  25%  Improvement  on  Low  Level  Difficulty  Exercises" 
TRY  METHOD  "Medium  Level  Difficulty"; 

END; 
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METHOD  "High  Level  Difficulty"; 

STEP  "Review  METT-T  Parameters", 

STEP  "Specify  High  Level  Parameters"; 

IF  "At  least  20%  Improvement  on  Medium  Level  Difficulty  Exercises" 
TRY  METHOD  "High  Level  Difficulty"; 

IF  "At  Least  50%  Improvement  over  Low  Level  Difficulty" 

AND  "Initial  Lack  of  Familiarity  with  Training  Device" 

TRY  METHOD  "High  Level  Difficulty"; 

END; 

END; 

RULE  "Select  Scenarios"; 

METHOD  "By  Weight  Method"; 

STEP  "Examine  Current  Rankings", 

STEP  "Choose  Tasks  to  Train", 

STEP  "Match  Selected  Tasks  against  Library  Scenarios"; 

IF  "Improve  Current  Training  Deficiencies" 

TRY  METHOD  "By  Weight  Method"; 

END; 

METHOD  "Group  by  Tactical  Category"; 

STEP  "Determine  Training  Event  Objectives", 

STEP  "Match  Objectives  with  Tactical  Category", 
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STEP  "Link  Tactical  Category  Tasks  to  Library  Scenario"; 
IF  "Preparation  for  a  Specific  Mission  is  Essential" 

TRY  METHOD  "Group  by  Tactical  Category"; 

IF  "Tasks  Logically  Grouped  into  Specific  Categories" 

TRY  METHOD  "Group  by  Tactical  Category"; 

END; 

END; 


RULE  "Edit  Scenarios"; 

METHOD  INCOMPLETE  "Edit  Scenarios"; 

STEP  GROUP  "Review/edit  Execution  Matrix", 
STEP  GROUP  "Review/edit  Operations  Overlay", 
STEP  "Review/edit  Operations  Order"; 

END; 

END; 


RULE  "Check  TADSS  Constraints"; 

METHOD  INCOMPLETE  "Check  TADSS  Constraints"; 

STEP  "Identify  Specific  TADSS", 

STEP  "Conduct  Constraint-based  Allocation  of  Resources", 
STEP  "Incorporate  Trainability  Factors"; 

END; 

END; 
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RULE  "Build  Input  Files  &  Exercise  Products"; 

METHOD  INCOMPLETE  "Build  Exercise  Products"; 
STEP  "Review  Input  Files", 

STEP  "Generate  Exercise  Output  Products"; 

END; 

END; 

RULE  "Capture  Assessments"; 

METHOD  "Exercise  Playback"; 

STEP  "Replay  exercise", 

STEP  "Assess  and  Explain  Areas  of  Concern", 
STEP  "Catalog/save  Replay  Files  for  Future  AARs" 
IF  "Time  Available  for  AAR  is  >  2  hrs" 

TRY  METHOD  "Exercise  Playback"; 

END; 

METHOD  "Review  Performance  Measures"; 

STEP  "Evaluate  Measures  of  Effectiveness", 

STEP  "Summarize  MOE  for  Follow-up  Training", 
STEP  "Record/print  Measures  and  Summary"; 

IF  "Unit  Commander  Desires  Analytical  Feedback" 
TRY  METHOD  "Review  Performance  Measures"; 
END; 
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METHOD  "Observer-controller"; 


STEP  "Tracks  the  scenario's  Main  Events", 

STEP  "Records  Observations  by  BOS  Areas", 

STEP  "Presents  Recommendations  for  Improvement", 
STEP  "Prepares  Written  Report"; 

IF  "An  OC  was  on-site  at  Simulation" 

TRY  METHOD  "Observer-controller"; 

END; 

END; 

EXCEPTION; 

END; 
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The  Training  Value  System 
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The  modified  TVS  model  proposed  by  Fitz-enz  (1994)  consists  of  the  following 

actions: 

a.  Pre-training  status  of  a  team  is  examined  by: 

(1)  identifying  command  training  objectives  and  criterion  for  success; 

(2)  selecting  a  target  traming  audience  and  level; 

(3)  choosing  tasks  which  are  essential  for  mission  success; 

(4)  prioritizing  tasks  based  on  training  proficiency  and  performance;  and 

(5)  establishing  the  need  for  a  gain  in  performance  by  the  assessment  of  the 
team’s  current  training  proficiency  through  a  pre-test. 

b.  Training  analysis  is  a  two-step  process  involving: 

(1)  problem  diagnosis;  and 

(2)  training  description  of: 

a)  the  need  for  training  on  a  set  of  tasks; 

b)  doctrinally-correct  tactical  scenarios; 

c)  the  applicable  training  devices  and/or  simulations;  anH 

d)  available  training  resources. 

c.  Training  event  selection  and  assessment  consists  of: 

(1)  choosing  a  live  field  training  event,  constructive  simulation-driven  exercise  or 
virtual  battlefield  exercise; 

(2)  identifying  the  variables  which  effected  performance;  and 

(3)  determimng  how  training  changed  performance. 

d.  Final  training  effectiveness  is  assessed  by: 

(1)  measimng  the  new  training  proficiency  level  and  then  calculating  the 
effectiveness  of  training  due  to  differences  in  quality,  productivity  and 
capability  to  execute  missions  between  the  pre-test  and  the  final  test; 

(2)  deciding  if  additional  training  will  be  required;  and 
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(3)  comparing  against  similar  simulation-based  training  exercises  (if  no  pre-test 
was  conducted). 
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Free  Recall  and  Position  Memory  Forms 
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Free  Recall  Memory  Form 


Subjects  drew  or  outlined  their  conceptual  understanding  of  the  training  planning 
process.  This  information  served  to  demonstrate  whether  subjects  were  able  to  increase 
their  imderstanding  of  the  training  planning  development  process.  The  diagrams  are 
pictorially  represented  and  kept  in  the  author’s  experimental  data  collection. 


Build  a  Diagram,  Picture,  or  Flow  Chart 
to  Show  the  Components  of  a 
Training  Planning  Model  for 
Simulation-based  Training 


Training  Planning  for  Simulations—  Position  Memory  Testing 


(Solution  Sheet  to  determine  ARC  values) 


Events:  When  to  Train 

Tasks:  What  to  Train 

Scenarios:  How  to  Train 

9 

2 

11 

3 

8 

10 

1 

7 

4 

5 

6 

In  a  Logical  Order,  fill  in  the  above  categories  with  the  components  of  a 
training  planning  model 

Use  the  number  to  indicate  position  on  the  chart  above: 

1.  Allocate  Tasks  to  Events 

2.  Assess  Training  Proficiency 

3.  Build  Calendars  &  Schedules 

4.  Build  Input  Files  &  Exercise  Products 

5.  Capture  Assessments 

6.  Check  TADSS  Constraints 

7.  Edit  Scenarios 

8.  Prioritize  Tasks 

9.  Record  METL 

10.  Select  Scenarios 

11.  Select  Training  Conditions 
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Appendix  H 


Free  Recall  Adjusted  Ratio  of  Clustering  Data 
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ARC  Value  Results  (from  Free  Recall  Testing) 


Automated  Manual 

Teams  Teams 


ID# 

Pretest 

Posttest 

ID# 

Pretest 

Posttest 

AOll 

-0.10 

-0.10 

B021 

0.12 

-0.32 

A041 

-0.32 

0.56 

B031 

0.12 

-0.10 

A042 

0.34 

0.12 

B061 

-0.10 

0.12 

A051 

-0.76 

-0.10 

B062 

-0.32 

-0.54 

A121 

-0.76 

-0.10 

Bill 

0.34 

1.00 

A122 

-0.10 

0.56 

B131 

0.12 

-0.10 

A161 

-0.32 

-0.10 

B132 

-0.10 

-0.32 

A171 

-0.10 

0.12 

B152 

-0.32 

-0.32 

A201 

-0.32 

0.34 

B181 

0.12 

-0.32 

A221 

-0.10 

0.34 

B182 

-0.32 

-0.32 

A241 

-0.32 

-0.10 

B211 

-0.32 

-0.76 

A311 

0.12 

0.34 

B212 

0.12 

0.12 

A3 12 

-0.10 

0.12 

B231 

-0.10 

0.56 

A351 

0.34 

0.78 

B251 

.  -0.10 

-0.32 

A361 

-0.76 

-0.76 

B301 

0.12 

0.12 

A362 

-0.76 

-0.32 

B302 

-0.10 

-0.10 

A401 

-0.32 

-0.10 

B331 

-0.32 

0.56 

A421 

-0.10 

0.34 

B332 

0.34 

0.12 
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Automated 

Pretest 

Posttest 

Mantial 

Pretest 

Posttest 

A431 

-0.54 

-0.32 

B341 

-0.54 

O 

c> 

1 

A432 

0.34 

0.12 

B411 

0.12 

-0.32 

A451 

0.12 

0.56 

B441 

-0.32 

-0.10 

A511 

-0.32 

-0.10 

B551 

0.12 

0.12 

A512 

-0.10 

-0.32 

B552 

-0.54 

0.78 

A521 

-0.10 

0.78 

B561 

0.12 

0.12 

A522 

0.34 

0.12 

B562 

-0.32 

0.12 

A531 

0.12 

0.12 

B571 

-0.10 

0.78 

A532 

0.12 

0.56 

B572 

-0.10 

0.56 

A541 

-0.32 

-0.32 

B581 

-0.10 

-0.10 

A542 

0.34 

0.12 

B582 

0.56 

0.34 

A611 

0.12 

-0.10 

B651 

-0.32 

0.12 

A621 

-0.54 

-0.10 

B652 

-0.54 

-0.10 

A622 

-0.10 

0.34 

B661 

-0.10 

-0.76 

A631 

-0.54 

-0.32 

B671 

0.12 

0.12 

A711 

-0.10 

0.12 

B672 

-0.32 

0.12 

A712 

0.12 

0.12 

B751 

-0.10 

-0.32 

A721 

-0.76 

0.12 

B752 

-0.76 

-0.54 

A731 

-0.10 

0.12 

B761 

-0.10 

-0.10 

A732 

-0.32 

0.12 

B762 

-0.10 

-0.32 

A741 

0.56 

0.12 

.  B771 

-0.76 

-0.32 
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Appendix  I 


Position  Memory  ARC  Data 
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ARC  Value  Results  (from  Position  Memory  Testing) 


Automated  Teams 

ID# 

Pretest 

Posttest 

AOll 

0.12 

0.34 

A012 

-0.32 

♦ 

A041 

-0.10 

0.12 

A042 

-0.54 

0.56 

A051 

-0.10 

0.34 

A052 

-0.76 

* 

A121 

-0.10 

0.12 

A\22 

-0.32 

-0.32 

A161 

0.12 

0.12 

A162 

0.56 

0.56 

A171 

0.34 

♦ 

A172 

-0.76 

* 

A201 

0.34 

-0.10 

A202 

0.12 

-0.32 

A221 

0.12 

♦ 

A222 

0.56 

0.34 

A241 

-0.76 

-0.10 

.A242 

0.12 

0.78 

A311 

0.34 

0.34 

Manual  Teams 


ID# 

Pretest 

Posttest 

B021 

0.34 

-0.76 

B022 

-0.54 

♦ 

B031 

O 

o 

1 

-0.10 

B032 

0.12 

♦ 

B061 

-0.76 

-0.54 

B062 

-0.76 

0.34 

Bill 

0.34 

-0.10 

B131 

-0.10 

0.34 

B132 

0.12 

-0.54 

B152 

-0.10 

0.12 

B181 

0.12 

0.34 

B182 

-0.32 

0.34 

B211 

-0.32 

-0.32 

B212 

0.34 

0.12 

B231 

-0.10 

0.56 

B251 

0.78 

0.56 

B252 

0.56 

0.34 

B301 

-0.32 

♦ 

B302 

0.78 

0.34 
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data  were  untisable  in  these  cases. 
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Appendix  J 


Quality  Rating  Form 
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User  Qualities  Rating  Scale 

Follow  the  Flow  Chart  to  Assess 

the  Adequacy  of  the  Method  you  used for 

Training  Planning  (Circle  the  number) 


Method  Characteristics:  Demands  on  the  user 
_ in  operating  system  , 

Excellent:  User  could  understand  and  execute 

Good:  Negligible  deficiencies  in  method 

Fair:  Mildly  unpleasant  deficiencies  in  method  I  I 


Is  method 

satisfactory  without 
^improvement?/' 


Deficiencies 

Minor,  but  somewhat  annoying  deficiencies 

A 

4 

Warrant  ^ 

Improvement 

Definitely  annoying  deficiencies 

5 

Annoying,  yet  tolderable  deficiencies 

6  > 

adequate\^ 

performance 

va^inable  with  a  tolerably 
user  workload?./^ 


Deficiencies 

Require 

Improvement 


^ajor  Deficiencies:  Cannot  adequatel  use 

_ without  making  adjustments 

Major  Deficiencies:  Considerable  adjustment 

_ needed  to  usemethod _ 

Major  Deficiencies:  Time  to  consider  using 
V _ alternate  methods  for  planning 


nethod  Controllable?^ 


Improvement 

Mandatory 


Major  Deficiencies:  Time  to  revert  to  other  jq 

_ methods  for  training  planning _ 


Training  Planning  Decisions 


From  Cooper  and  Harper,  (1969) 
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Quality  Rating  Data 
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Quality  Ratings  by  Teams 
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Appendix  L 


Simulation  MOE  Definition  and  Form 


Measures  of  Effectiveness: 


MOE  #1:  Number  of  Tanks  (Mis)  which  survived: _ 

MOE  #2:  Number  of  Mechanized  Infantry  vehicles  (M2s)  which  survived: _ 

MOE  #3:  Number  of  Enemy  ground forces  killed: _ 

Record  of  Tasks  trained  in  Janus:  (Use  for  Sessions  #i  &  #4) 

Proficiency  of  Training  (this  data  were  collected,  but  not  used  in  the  analysis  for  the 


hypothesis  or  study  conclusions) 


Task# 

Training  Task 

Initial 

Final 
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Simulation  MOE  Data 


Results  of  Janus  Simulation  Iterations  by  Teams* 


*This  is  one  part  of  the  study  where  the  data  were  not  collected  well  due  to  users 
not  following  directions  to  ensure  that  all  forms  were  filled  out  completely.  In  some 
cases,  the  simulation  exercise  was  closed  out  too  soon  to  enable  collection  of  this  data. 
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Appendix  N 


Task  Load  Index  Scores  Form 
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TASK  LOAD  INDEX  RATING  SHEET 


Team  # 


Session  # 


Date 


All  responses  wiil  be  kept  in  strict  confidence.  Only  aggregate  information  will  be  reported.  Please 
answer  the  following  questions  by  placing  a  check  in  the  appropriate  interval. 

1 .  Mental  Demand:  How  much  thinking,  deciding,  searching,  remembering  or  looking  was  required  in 
the  previous  session? 


Low  High 

2.  Physical  Demand:  How  much  physical  activity  was  required? 


Low  High 

3.  Temporal  Demand:  How  much  time  pressure  did  you  feel  due  to  the  rate  or  pace  at  which  the  tasks  or 
task  elements  occurred?  Was  the  pace  slow  and  leisurely  or  rapid  and  frantic? 


Low  High 

4.  Performance:  How  successful  do  you  think  you  were  in  accomplishing  the  goals  of  the  session  set  by 
the  experimenter? 


Good  Poor 

5.  Effort:  How  hard  did  you  have  to  work  (mentally  and  physically)  to  accomplish  your  level  of 
performance? 


Low  High 

6.  Frustration:  How  discouraged,  irritated,  stressed  and  annoyed  versus  secure,  gratified,  content  and 
relaxed  did  you  feel  during  the  session? 


Adapted  from  Hart  and  Staveland  (1988) 


178 


Appendix  O 


Task  Load  Index  Scores  Data 
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Manual  Team  Task  Load  Index  Scores  by  Factor  rwith  weighting^ 


TM# 

Mental 

Demand 

Physical 

Demand 

Temporal 

Demand 

Perform 

ance 

Effort 

Frustration 

SUM 

B2 

43.00 

4.83 

54.00 

44.00 

49.50 

19.67 

215.00 

B3 

56.58 

8.33 

38.50 

12.75 

52.50 

39.58 

208.25 

B6 

47.50 

1.67 

31.00 

32.67 

34.00 

49.58 

196.42 

Bll 

60.00 

2.08 

99.75 

35.50 

43.75 

17.00 

258.08 

B13 

58.33 

0.00 

21.33 

33.25 

79.17 

10.17 

202.25 

B15 

70.83 

0.00 

21.67 

36.00 

21.67 

11.33 

161.50 

B18 

81.75 

0.00 

30.00 

36.67 

47.00 

19.75 

215.17 

B21 

35.42 

0.00 

12.00 

12.67 

39.00 

27.42 

126.50 

B23 

56.00 

0.00 

18.00 

13.00 

35.83 

4.83 

127.67 

B25 

60.67 

0.00 

26.50 

44.00 

34.00 

10.67 

175.83 

B30 

79.33 

4.17 

48.50 

21.50 

45.83 

29.33 

.  228.67 

B33 

47.25 

1.08 

22.67 

32.67 

26.75 

49.33 

179.75 

B34 

57.75 

1.50 

10.67 

22.67 

78.00 

16.50 

187.08 

B41 

42.00 

2.83 

32.08 

39.67 

51.75 

0.00 

168.33 

B44 

42.50 

0.00 

14.17 

73.33 

41.50 

46.00 

217.50 

B55 

67.17 

0.00 

22.50 

23.83 

42.83 

49.17 

205.50 

B56 

90.50 

2.33 

35.00 

19.00 

58.58 

16.83 

222.25 

B57 

1 

25.50 

1.17 

8.92 

21.67 

23.33 

5.25 

85.83 

B58 

72.67 

1.08 

54.67 

22.00 

54.50 

21.00 

225.92 

TM# 

Mental 

Demand 

Physical 

Demand 

Temporal 

Demand 

Perform 

ance 

Effort 

Frustration 

SUM 

B65 

58.17 

0.00 

30.42 

35.17 

32.00 

11.67 

167.42 

B66 

60.00 

6.33 

23.25 

16.75 

36.00 

22.83 

165.17 

B67 

50.67 

3.00 

23.67 

23.50 

44.67 

14.00 

159.50 

B75 

43.00 

0.00 

5.83 

16.67 

60.00 

18.67 

144.17 

B76 

87.50 

7.00 

35.00 

27.00 

55.00 

0.00 

211.50 

B77 

30.50 

0.00 

34.67 

5.14 

19.49 

43.46 

133.26 

8 


TM# 

Mental 

Demand 

Physical 

Demand 

Temporal 

Demand 

Perform 

ance 

Effort 

Frustration 

SUM 

A1 

84.17 

0.00 

27.00 

34.50 

28.67 

27.67 

202.00 

A4 

32.00 

0.00 

40.83 

55.00 

46.50 

16.00 

190.33 

A5 

43.00 

11.83 

23.33 

47.33 

71.67 

0.00 

197.17 

A12 

35.58 

1.92 

11.33 

26.25 

23.00 

12.75 

110.83 

A14 

41.00 

0.00 

46.67 

17.00 

23.00 

24.50 

152.17 

A16 

75.00 

0.00 

54.67 

45.50 

12.33 

24.00 

211.50 

A17 

44.58 

4.42 

28.00 

21.50 

57.33 

64.50 

220.33 

A20 

54.00 

0.00 

14.50 

20.00 

44.33 

29.50 

162.33 

A22 

47.83 

2.08 

28.33 

43.33 

39.00 

6.08 

166.67 

A24 

48.00 

0.00 

14.00 

18.00 

95.00 

69.33 

244.33 

A31 

52.50 

5.33 

31.50 

40.67 

27.92 

4.42 

162.33 

A36 

45.00 

1.67 

37.00 

49.33 

21.00 

36.67 

190.67 

A40 

76.67 

3.33 

30.50 

41.50 

33.00 

0.00 

185.00 

A42 

32.00 

0.00 

23.50 

61.67 

8.83 

149.00 

A43 

34.67 

2.17 

9.50 

30.83 

17.00 

12.00 

106.17 

A45 

37.33 

0.00 

58.33 

27.00 

10.67 

13.33 

146.67 

A51 

44.00 

0.00 

21.50 

42.67 

21.00 

8.50 

137.67 

A52 

57.75 

0.00 

22.92 

16.33 

32.50 

27.00 

156.50 

A53 

48.00 

0.00 

25.42 

46.50 

44.67 

12.33 

176.92 
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Appendix  P 


Performance  Times  Form 
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Training  Planning  Practical  Exercise  Assignment  &  Time  Sheet 


•  Team  Assignment:  You  are  the  team  leader  for  Team  # _ 

•  Please  take  a  few  minutes  to  ensure  your  team  understands  the  instructions  for 
the  Practical  Exercise.  Brief  your  group  leader  (MAJ  Stone  or  Dr.  Crissey)  at  the  end  of 
each  session  to  ensure  you  have  met  the  session  requirements,  in  order  to  proceed  to  the 
next  phase  of  the  exercise. 

Results  will  be  aggregated  to  determine  total  and  average  times  by  group. 


Please  ensure  you  be  as  accurate  as  possible  in  recording  your  times.  Once  you 
have  completed  a  Session,  take  a  break  and  then  return  to  begin  the  next  session. 


# 

Session 

Start 

Time 

End 

Time 

Total  Minutes 

1 

Analyzing  and  Prioritizing 
Tasks 

— 

2 

Building  Training 
Simulation  Scenario 

• 

3 

Preparing  &  Running  the 
Simulation 

• 

• 

— 

— 

4 

Assessing  Proficiency  fi-om 
the  Simulation  Exercise 

• 

• 

— 
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Perfonnance  Times  Data 
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Manual  Team  Times  by  Session 


TM# 

T1_M 

T2_M 

T3_M 

T4_M 

B2 

30 

80 

60 

30 

B3 

30 

90 

70 

35 

B6 

30 

100 

60 

30 

Bll 

30 

80 

120 

20 

B13 

30 

45 

85 

30 

B15 

20 

34 

105 

20 

B18 

45 

75 

115 

14 

B21 

25 

35 

120 

25 

B23 

21 

45 

115 

19 

B25 

25 

25 

140 

15 

B30 

32 

61 

98 

15 

B33 

27 

59 

160 

10 

B34 

30 

60 

120 

55 

B41 

40 

56 

96 

20 

B44 

50 

75 

70 

15 

B55 

72 

95 

175 

30 

B56 

55 

90 

170 

170 

B57 

43 

53 

225 

23 

B58 

51 

90 

195 

30 

JO' 

M 


200 


225 

220 

250 

190 

179 

249 

205 

200 

205 

206 

256 

265 

212 

210 

372 

485 

344 

366 


B65  70  95  200  30  395 

B66  75  90  175  20  360 

B67  50  122  122  22  316 

B75  53  70  105  22  250 

B76  67  85  180  30  362 

B77  47  86  105  45  283 

Avg:  41.92  71.84  127.44  31  272.2 


Automated  (TREDS)  Team  Times  by  Session 


TM# 

T1_A 

T2_A 

T3_A 

T4_A 

T_TOT 

A 

A1 

25 

135 

60 

15 

235 

A4 

28 

102 

35 

32 

197 

A5 

31 

120 

20 

37 

208 

A12 

35 

115 

50 

6 

206 

A14 

35 

68 

60 

13 

176 

A16 

60 

44 

90 

15 

209 

A17 

45 

53 

60 

13 

171 

A20 

18 

45 

35 

30 

128 

A22 

12 

no 

45 

35 

202 

A24 

30 

75 

90 

70 

265 

A31 

20 

90 

40 

15 

165 

A36 

OB 

75 

10 

235 

A40 

20 

138 

100 

35 

293 

A42 

16 

75 

90 

18 

199 

A43 

8 

76 

125 

15 

224 

A45 

20 

30 

200 

30 

280 

A51 

12 

no 

60 

15 

197 

A52 

!  9 

130 

120 

30 

289 

A53 

17 

no 

188 

10 

325 

189 


TM# 

T1  A 

T2  A 

T3  A 

T4  A 

T  TOT 

A 

A54 

7 

113 

192 

21 

333 

A61 

11 

144 

103 

25 

283 

A62 

11 

116 

50 

30 

207 

A63 

60 

no 

40 

60 

270 

A71 

45 

120 

120 

45 

330 

A72 

25 

125 

135 

45 

330 

A73 

20 

130 

118 

25 

293 

A74 

18 

127 

30 

25 

200 

Avg: 

24.741 

86.333 

26.667 

238.889 
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Demographics  Questionnaire  Data  and  Results 
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Demographics  Data  for  the  Manual  Teams—  Background  Information 


Team#  Position  Rank!  A 


Time  in  Time  in 


Service  Unit 
Cm)  (yrs) 


B2 

PLT  SGT 

E7 

47 

14.00 

1.00 

B2 

PLT  LDR 

02 

25 

3.00 

2.50 

B3 

SEC  LDR 

E6 

31 

9.00 

2.50 

B3 

XO 

02 

25 

2.00 

2.00 

B6 

SPTPL 

02 

26 

3.00 

2.25 

B6 

MED  PS 

E7 

37 

19.00 

2.17 

Bll 

ASST 

S3AIR 

01 

23 

0.67 

0.17 

Bll 

CMPTR 

OPS 

E4 

27 

4.00 

0.50 

B13 

CO  CDR 

03 

31 

9.00 

0.42 

B13 

TNGNCO 

E6 

31 

15.00 

3.00 

B15 

PLT  LDR 

01 

33 

6.00 

0.17 

B15 

OPS  NCO 

E6 

30 

11.00 

3.00 

B18 

E6 

38 

9.00 

2.00 

B18 

PLT  SGT 

E7 

33 

13.58 

0.06 

B21 

XO 

02 

26 

4.00 

1.17 

B21 

MG 

E6 

32 

10.00 

0.50 

B23 

PLT  LDR 

01 

23 

2.00 

1.50 

B23 

PLT  SGT 

E7 

37 

18.00 

3.00 

92 


B25 

XO 

02 

26 

4.00 

0.04 

B25 

1ST  SGT 

E8 

42 

22.00 

0.75 

B30 

PLTLDR 

01 

23 

1.00 

0.42 

B30 

PLT  SGT 

E7 

37 

15.00 

1.50 

B33 

BICC 

01 

25 

2.00 

0.83 

B33 

INTEL 

SPEC 

E4 

31 

8.50 

1.00 

B34 

PLTLDR 

01 

26 

5.00 

0.42 

B34 

TNGNCO 

E5 

24 

4.50 

0.02 

B41 

PLT  LDR 

02 

30 

8.00 

2.00 

B41 

PLT  SGT 

E7 

39 

18.00 

3.00 

B44 

XO 

02 

25 

3.00 

2.00 

B44 

PLT  LDR 

01 

25 

2.00 

1.50 
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Team# 

Position 

Rank 

Age 

Time  in 

Time  in 

Service 

Unit 

(yts) 

(yrs) 

B55 

CO  FSO 

01 

24 

0.67 

1.50 

B55 

PLT  SGT 

E5 

29 

9.00 

2.00 

B56 

PLTLDR 

01 

22 

1.00 

0.33 

B56 

PLT  SGT 

E7 

32 

12.00 

3.00 

B57 

XO 

02 

24 

2.00 

0.67 

B57 

PLT  SGT 

E7 

29 

11.80 

0.50 

B58 

BNS2 

03 

29 

5.75 

1.00 

B58 

PLTLDR 

02 

25 

6.00 

1.42 

B65 

XO 

02 

25 

3.50 

0.33 

B65 

PLT  LDR 

CDT 

22 

0.00 

0.04 

B66 

PLTLDR 

02 

26 

3.00 

1.17 

B66 

TANK 

CDR 

E6 

37 

19.00 

3.00 

B67 

PLTLDR 

01 

22 

1.00 

0.50 

B67 

PLT  SGT 

E6 

40 

15.00 

2.00 

B75 

PLT  LDR 

01 

26 

3.00 

0.08 

B75 

PLT  SGT 

E6 

32 

13.00 

1.50 

E5 

30 

12.00 

3.50 

B76 

TANK 

CDR 

E6 

35 

15.00 

0.25 
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B77 

TANK 

CDR 

E6 

29 

11.00 

0.08 

B77 

GUNNER 

E5 

23 

3.50 

3.25 
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Demographics  Data  for  the  Automated  Teams--  Background  Information 


Team# 

Position 

Rank 

Age 

Time  in 

Service 

Unit 

fYr§) 

(Yrs) 

A1 

MED  PL 

02 

25 

2.00 

2.00 

A1 

SEC  LDR 

E6 

32 

12.00 

0.02 

A4 

CO  CDR 

03 

29 

7.00 

0.92 

A4 

MORT  PL 

02 

24 

4.00 

1.50 

A4 

PLT  SGT 

E7 

31 

11.00 

1.00 

A5 

SQD  LDR 

E5 

30 

7.00 

0.33 

A5 

TNGNCO 

E5 

29 

9.00 

0.42 

A12 

S3  AIR 

03 

30 

9.00 

0.33 

A12 

TNG 

CLERK 

E4 

24 

3.00 

1.00 

A14 

PLT  SGT 

E6 

29 

11.00 

1.00 

A14 

27 

4.50 

0.42 

A16 

TANK 

CDR 

E6 

33 

12.00 

0.08 

A16 

PLANS 

OFF 

02 

27 

! 

3.00 

0.50 

A17 

SCOUT 

PL 

01 

23 

1.00 

0.33 

A17 

HQ  TANK 

E7 

43 

19.00 

0.02 

A20 

CO  CDR 

03 

32 

6.00 

0.33 

A20 

INTEL 

E5 

25 

8.00 

0.50 
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NCO 


A22 

PLT  LDR 

01 

33 

4.00 

1.00 

A22 

PLT  SGT 

E7 

34 

16.00 

1.75 

A24 

PLT  LDR 

01 

30 

5.00 

0.58 

A24 

PLT  SGT 

E7 

34 

15.00 

2.00 

A31 

TANK 

CDR 

E6 

32 

12.00 

2.00 

A31 

CO  CDR 

03 

29 

7.00 

0.25 

A36 

PLT  LDR 

01 

24 

1.00 

0.42 

A36 

PLT  SGT 

E6 

30 

15.00 

1.00 

A40 

CO  CDR 

03 

36 

12.00 

0.50 

A40 

TANK 

CDR 

E6 

27 

8.00 

0.42 

A42 

PLT  LDR 

02 

24 

2.00 

0.33 

A42 

PLT  SGT 

E7 

32 

15.00 

1.50 

A43 

TRNG 

OFF 

03 

27 

6.00 

2.00 

A43 

BMO 

02 

25 

3.00 

2.00 

A45 

S3 

04 

37 

12.00 

0.50 

A45 

TANK 

CDR 

E6 

32 

11.00 

0.67 

Team# 

Position 

Rank 

Age 

Time  in 

Time  in 

Service 

Unit 

{j2S} 

A51 

CO  CDR 

03 

31 

6.50 

1.17 

A51 

PLT  SGT 

E6 

35 

16.00 

4.00 

01 

28 

9.00 

0.33 

A52 

PLTLDR 

02 

25 

2.00 

0.50 

A53 

PLTLDR 

01 

22 

1.00 

0.33 

A53 

PLTLDR 

01 

23 

1.00 

0.67 

A54 

PLTLDR 

01 

22 

1.00 

0.50 

TANK  E5  30  7.00  2.00 

CDR 

S3  SGM  E9  42  22.00  1.00 

ISTSGT  E8  40  19.00  3.50 

CO  CDR  03  28  5.00  0.25 

PLTLDR  CDT  20  0.00  0.04 

GUNNER  E5  23  5.00  0.50 

CO  FSO  02  23  2.00  0.33 

TANK  E6  34  10.00  1.50 

CDR 


CDT 

21 

0.00 

0.04 

A72 

CO  FSO 

01 

23 

1.08 

0.25 

A72 

TANK 

E6 

31 

1  11.00 

3.00 

A54 

A61 


A63 
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CDR 

A73 

CO  CDR 

03 

26 

5.00 

0.08 

A73 

GUNNER 

E5 

27 

4.50 

0.42 

A74 

BN  NBC 
NCO 

E6 

30 

12.00 

3.00 
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Appendix  S 


Post-Exercise  Questionnaire  Data  and  Feedback 
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Post-Exercise  Questionnaire  Statistical  Results 

Knowledge  Increase 
Mann- Whitney  Test 

KI_M  (Knowledge  Increase_Manual  Teams)' 

'KI_A  (Knowledge  IncreaseAutomated  Teams) 

KI_M  N=25  Median  =  2.5000 

KI_A  N=27  Median  =  3.0000 

Point  estimate  for  ETA1-ETA2  is  0.0000 

95.2  Percent  C.I.  for  ETA1-ETA2  is  (-0.5000,0.0001) 

W  =  601.5 

Testof  ETAl  =ETA2  vs.  ETAl  <ETA2  is  significant  at  0.1339 
The  test  is  significant  at  0.1254  (adjusted  for  ties),  showings  a  87.5%  confidence 
that  the  automated  teams  increased  their  knowledge  of  the  process  over  the  manual  teams 
based  on  a  4-point  Likert  scale  with  4  being  the  most  extensive  increase  in  Knowledge. 
These  results  are  not  significant  at  the  study’s  level  of  5%. 
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Open-ended  Questions 


5.  WHAT  AREAS  DO  YOU  CONSIDER  NEED  IMPROVEMENT  IN  THE 
TRAINING  PLANNING  PROCESS? 

♦  Need  more  experience  first. 

♦  The  amount  of  time  it  takes  to  ID  task  and  sub-task. 

♦  The  system  needs  to  be  streamlined  somewhat.  Recognizing  it  is  brand  new 
to  us,  it  was  not  as  user  friendly  as  it  could  have  been.  Direction  in  each 
window  or  window  "hints"  could  greatly  help  speed  us  up. 

♦  Due  to  limitations  here  at  Ft  Knox,  more  time  is  needed  for  the  running  of  the 
scenarios. 

♦  None.  It  is  very  thought  out,  and  by  the  book,  excellent  program. 

♦  This  training  is  best  suited  for  a  level  higher  than  PLT. 

♦  Software  needs  to  be  fine  tuned. 

♦  Better  blocks  of  instructions. 

♦  Different  material  and  tasks. 

♦  How  we  train  via  increased  creativity  to  achieve  training  standards. 

♦  Longer  time  to  work  with  OP  orders. 

♦  Resource  management. 
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♦  Area  where  you  could  research  training. 

♦  Input  from  xmit  to  be  trained. 

♦  Program  is  very  well  put  together.  There  are  no  specific  areas  that  need 
particular  improvements,  but  I  would  like  more  time  on  the  "hands-on" 
training. 

♦  Too  many  windows  to  deal  with  on  programs  (i.e.,  easier  if  some  lists  were 
printed) 

♦  The  planning  process  works  very  well  with  TREDS.  However,  the  training 
process  needs  to  incorporate  both  simulation  and  real  life  training  (i.e.,  an 
actual  OPORD  briefed  and  graded  by  MTP  in  conjunction  with  the 
simulations).  Simulations  are  not  a  stand-alone  item. 

♦  Increased  interaction  with  supporting/other  units  and  higher  HQ  on  Janus. 

♦  Something  to  list  needed  material  for  training  being  planned. 

♦  Individual  and  squad  level  tasks. 

♦  I  felt  like  we  went  straight  into  "run"  with  this  exercise.  What  happened  to 
crawl  and  walk? 

♦  Bug  in  the  database  system. 

♦  As  the  "B"  team  our  method  is  the  same  method  that  I  have  used  since  1 986. 
These  questions  focus  on  the  "A"  team  and  not  what  I  did  today. 

♦  The  orders  need  to  be  broken  down  and  explained,  and  how  they  are  put 
together. 

♦  Planning  to  cover  all  elements  of  the  task  your  are  training  on. 


203 


♦  Better  understanding  of  the  training  objective/walked  into  class  with  no  idea 
what  was  going  to  happen. 

♦  The  level  at  which  success  or  failure  are  met. 

♦  Understanding  all  the  applicable  icons  that  enhance  the  overall  system. 

♦  Time  to  train. 

♦  More  flexibility  to  be  able  to  drive  the  process  instead  of  process  being 
dictated  to  you. 

♦  Just  big  ironing. 

♦  Having  a  clear  focus  of  what  tasks  need  training.  Be  more  realistic  about  what 
benefits  may  be  derived  from  Janus. 

♦  A  little  more  user  friendly  for  inexperienced  computer  operators. 

♦  The  current  planning  process  is  sometimes  tedious. 

♦  More  imderstanding  of  each  individual  task. 

♦  Proper  standardized  assessment. 

♦  Minor  software  glitches. 

♦  Proper  identification  of  "to  be  trained  tasks" . 

♦  Integration  of  slice  elements. 

♦  Reduce  repetition  of  information  in  oporders. 
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♦  Need  more  time. 


♦  A  prior  knowledge  of  the  steps  involved. 

♦  An  honest  assessment  of  where  you  stand. 

♦  Once  training  is  planned  -  we  need  to  be  able  to  carry  it  out  rather  than 
canceling  it  because  of  vmexpected  stupid  details. 

♦  Time  to  leam  MS  Word,  MS  Power  Point.  Contrary  to  popular  belief,  these 
programs  have  not  been  fielded  below  BN  level. 

♦  Very  little  improvement  needed. 

♦  Is  good  from  tank  commander  position. 

♦  Easier  access  to  hard  copies.  Try  to  put  more  info  on  less  separate  lists  and 
files. 

♦  How  the  METL  is  linked  up  to  the  task. 

♦  Time  to  conduct  realistic  training. 

♦  Time. 

♦  I  think  it  would  be  better  if  you  came  up  with  your  plan  without  the  computer, 
then  use  the  computer  to  see  how  effective  your  plan  was.  (Not  adjust 
vehicles  when  on  computer) 

♦  The  process  integration  of  getting  into  the  program,  misinformation,  and 
misunderstanding  of  reason  of  purpose. 

♦  Computer  skills. 
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Shorter  training  periods. 


♦  Issue  operation  orders/execution. 

♦  Develop  all  configurations  or  train  distracters  that  may  occur  and  try  to  avoid 
them.  Implement  more  Top  Down  oriented  training  platform. 

♦  Scheduling  of  training  and  utilization  of  resources. 

♦  Possibly  additional  hands  on  for  computer.  As  well  as,  a  thorough  knowledge 
of  expectations  prior  to  arrival. 

♦  Determining  task  to  be  trained  on,  then  prioritizing  those  tasks. 

♦  The  area  of  planning,  preparing  and  writing  an  operations  order,  so  that  it  is 
simplified  for  junior  leaders. 

♦  Make  the  computers  more  ragged. 

♦  Try  to  bring  this  to  an  Army  intellect  level,  i.e.,  this  is  presented  like  the 
serious,  academic  design  it  is,  but  will  need  to  be  understood  and  used  by 
people  with  much  less  academic  training  and  confidence. 

♦  Fewer  sub-task  listings  in  the  ARTEP.  Examples  of  what  qualify  T,  P,  or  U. 

♦  As  much  as  the  METL  should  drive  training,  there  are  many  other  missions 
we  must  accomplish,  maintenance,  red  cycle,  etc.,  and  I  did  not  see  a  way  to 
plan  for  that.  Additionally,  with  the  constant  flux  in  missions,  especially 
OOW,  having  an  MTP  driven  training  plan  can  lead  an  outside  observer  to 
believe  that  there  is  not  training  going  on.  i.e.,  1-66  goes  to  Cuba  to  guard 
Haitians  and  does  no  METL  training  for  3  months  because  riot  suppression  is 
not  in  the  MTP. 
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♦  Obtaining  time  and  training  resources.  Assessing  unit  proficiency  and 
retraining. 

♦  The  graphics  (overlay)  process  is  difficult  to  use.  It  is  difldcult  to  use  a  map 
on  a  screen.  It  is  much  easier  to  draw  the  graphics  by  hand.  Eventually  the 
computer  graphics  will  have  to  be  copied. 

♦  Consideration  for  training  distracters. 

6.  WHAT  DID  YOU  LIKE  MOST  ABOUT  TfflS  EXERCISE? 

♦  Using  the  Windows  environment  -  already  familiar  with  it. 

♦  It  closely  matched  training  management  system  in  organization  but  much 
easier  than  current  methods. 

♦  Phase  III  -  when  you  were  able  to  execute  y  om  plan  and  see  how  it  worked. 

♦  OPORD  was  very  easy  to  modify.  Janus  capabilities. 

♦  It's  a  fairly  easy  system  to  understand. 

♦  Easier  to  find  tasks.  Ability  to  print  what  you  need.  Quick  access  to  training 
status. 

♦  Being  able  to  see  the  plan  executed  as  a  third  party  (i.e.,  not  in  a  turret 
focusing  on  a  small  portion  of  the  battlefield). 

♦  Easy  to  work  with  and  obtainable. 

♦  The  Janus  exercise. 
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♦  Application  to  Janus. 

♦  I  enjoyed  actually  planning  the  mission. 

♦  The  Amiy's  idea  of  expanding. 

♦  The  information  was  known  already. 

♦  Janus  execution. 

♦  Everything. 

♦  Management  of  METL  task  proficiency  automated  cross  walk. 

♦  The  actual  Janus  battle. 

♦  I  finally  got  involved  on  what  to  train  and  how  to  do  it  instead  of  being  told. 

♦  I  liked  being  able  to  conduct  a  training  scenario  and  observing  my  strengths 
and  weaknesses  in  deploying  my  forces  and  establishing  defend  positions. 

♦  Getting  hands-on  with  Janus. 

♦  It  was  much  more  user  friendly  that  trying  to  do  it  "stubby  pencil". 

♦  Almost  instantaneous  feed  back  on  capabilities  and  planning  execution. 

♦  Execution. 

♦  Watching  defensive  plan  come  to  fruition. 
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♦  It  was  good  to  be  able  to  see  what  areas  you  need  to  train  for  each  anH 
how  to  incorporate  those  tasks  into  training. 

♦  The  actual  Janus  time. 

♦  Allows  unit  to  train  as  you  would  like  to  fight  and  review  results  with  data 
measurement. 

♦  The  actual  running  of  mission  to  the  computer. 

♦  This  exercise  gave  feedback  on  the  results  of  your  planning.  This  is  valuable 
knowledge  before  going  out  to  the  field  to  actually  do  ARTEP  missions. 

♦  Small  group  instruction. 

♦  Evaluates  weaknesses  and  able  to  work  the  order  for  these  tasks. 

♦  The  planning  process,  and  how  it  came  to  play  with  computer  technology. 

♦  The  ability  on  the  same  system  to  determine  what  needs  to  be  trained,  put  it  on 
a  calendar  and  write  an  order  with  overlays  to  execute.  Janus  or  other  system 
don't  need  to  be  a  part  of  this  -  a  unit  would  benefit  just  working  off  a  terrain 
board. 

♦  Actual  hand-on  to  see  if  we  trained  the  tasks. 

♦  The  pre-written  docximents  and  METL  breakdown. 

♦  Access  to  knowledge.  Ease  of  use. 

♦  Nothing. 

♦  New  and  useful. 
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♦  Practice  doing  the  training  planning  process. 

♦  The  ability  to  train  faster  with  more  input. 

♦  Simulation  demonstration. 

♦  TREDS  takes  information  from  many  different  sources  and  enters  it  into  a 
common  program.  Therefore,  once  you  do  an  assessment  of  your  proficiency 
in  your  METL  tasks,  TREDS  can  help  you  plan  and  direct  your  training  to 
address  weaknesses  and  sustain  strengths. 

♦  The  program  was  very  user  friendly. 

♦  Planning  at  the  company  level. 

♦  The  computer  gave  you  a  picture  to  work  with.  You  could  easily  try  different 
approaches  quickly. 

♦  Paints  a  better  picture  of  the  different  scenarios. 

♦  The  Janus  system  is  an  exciting  tool. 

♦  It  was  different. 

♦  Playing  with  computers. 

♦  The  computer  simulation  battle. 

♦  Janus  training  is  only  2  hours. 

♦  Having  subordinates  act  as  team  commander.  Relaxed,  helpful  approach  of 
trainers. 
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Planning. 


♦  I  was  interested  to  watch/control  the  battle  on  Janus. 

♦  Being  able  to  watch  the  whole  battlefield. 

♦  Learning  how  to  better  "sequence"  the  battlefield. 

♦  Force  on  Force  Exercise. 

♦  Realization  of  the  system. 

♦  You  can  see  the  results  of  your  plan,  this  shows  you  how  well  you  did  and 
increases  effort  with  the  planning  process. 

♦  Computer  movement  of  the  icons  and  actual  visual  benefits  of  seeing  plan 
work. 

♦  The  automatic  inputs  within  the  TREDS  system. 

♦  The  thought  process  going  into  the  assessment. 

♦  The  automated/fighting  of  the  battle. 

♦  Watching  the  actual  battle. 

♦  The  planning  phase  and  execution  of  the  mission. 

♦  The  automation  of  the  updates. 

♦  It  gave  you  a  great  amount  of  flexibility,  let  you  learn  valuable  mistakes  with 
little  loss. 
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♦  Introduction  to  Janus. 

♦  The  flexibility  and  user  friendliness  of  the  system. 

♦  The  hands-on  experience  utilizing  the  Janus  system. 

♦  Session  three. 

♦  The  execution  of  the  planning  process.  To  see  how  all  ofthe  hours  of  prep 
would  turn  out. 

♦  Determining  weaknesses  in  my  planning  process,  but  also,  able  to  see  how 
planning  process  can  be  effective. 

♦  Actually  being  able  to  watch,  observe  and  analyze  my  own  battle  plan  as  it 
happens. 

♦  Actually  executing  the  plan. 

♦  Janus  was  fun. 

♦  The  program  has  all  of  the  specific  tasks  -  and  is  pretty  easy  to  use.  You  don't 
have  to  search  around  through  a  bunch  of  manuals. 

♦  Introduction  to  Janus. 

♦  That  METL  did  drive  the  training.  METL  is  the  focus  of  everything  and 
unless  that  is  your  baseline,  you  easily  get  side  tracked  into  dog  and  pony 
missions. 

♦  Quickness  of  execution  -  able  to  implement  scenario  changes. 
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♦  The  automated  assessment  process  gave  easy  access  to  the  information  and 
made  the  storage  and  recall  a  useful  tool  for  training.  The  ability  to  put  an 
OPORD  on  the  system  is  also  useful. 

♦  Janus. 


7.  WHAT  WOULD  YOU  CHANGE  TO  MAKE  THIS  A  BETTER  EXERCISE? 

•  Have  the  entire  program  so  that  all  specialty  areas  can  benefit  more  and 
imderstand  it  in  their  level. 

•  Spend  more  time  during  familiarization.  I  would  make  certain  I  sent  people 
that  knew  Microsoft  and  were  familiar  with  Janus. 

•  Have  leaders  plan  at  their  level.  This  was  the  first  time  I  had  to  execute  a 
company  level  mission  I  planned. 

•  Explain  in  detail  during  familiarization  how  all  the  different  aspects  of  the 
system  tie  together. 

•  Planning  and  executing  the  training. 

•  Allow  more  time. 

•  I  don't  have  enough  knowledge  need  to  work  on  it  more. 

•  Can't  think  of  any. 

•  More  detailed  explanation  of  how  this  can  and  will  make  the  job  better  or 
easier. 

•  Spend  an  hour  or  so  to  be  familiar  with  Janus  in  order  to  execute  the  plan. 
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More  time.  This  is  not  a  one-day  affair. 


More  in-depth  operations  on  war  tactics. 

Allow  more  freedom  to  execute  developed  plan. 

More  time  on  OPORDS. 

Use  unit  training  calendars  and  events. 

Not  so  many  surveys. 

Give  more  time  to  prepare  and  conduct  the  exercise. 

Shorter  in  time  duration.  More  clearly  defined  goals. 

Nothing  in  particular. 

Have  some  info  printed  and  use  the  program  to  bring  up  METL  tasks. 

We  were  using  BN  level  training  but  trying  to  modify  it  to  a  company  plan. 
This  should  be,  and  can  be  tailored  for  the  individual  attempting  to  use  it. 

Interaction  with  adjacent  units  on  Janus. 

Spend  more  time  in  using  the  computer. 

Involve  more  military  personnel  on  your  staff  that  have  similar  experience. 
More  prep  time. 

Time  allowed  on  system  per  unit. 
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•  Make  sure  other  soldiers  understand  what  an  order  is  and  what  it  does  for  the 
Task  force, 

•  Bring  an  exercise  down  to  the  PLT  level,  so  platoon  tasks  can  be  worked  on, 
in  order  to  build  up  to  completing  and  being  fully  trained  on  company  tasks. 

•  Less  outside  distractions  (Location!). 

•  Nothing. 

•  More  time.  More  role  playing  at  other  terminals. 

•  Enjoyed  it.  More  time  to  learn  the  system. 

•  Have  more  time  to  get  better  at  system. 

•  Increase  database. 

•  Better  orientation  to  the  purpose  of  the  exercise.  Better  prior  planning  to 
allow  the  "guinea  pigs"  to  prepare. 

•  The  CSS  needs  a  part  in  the  exercise. 

•  More  time  with  the  equipment. 

•  Make  the  OPORD  readable. 

•  A  complete  overview  and  run  through  demonstration  before  starting  the 
exercises.  A  progress  assessment  at  each  Phase  of  the  exercise  during  which 
each  team  leader  is  required  to  brief  his  group's  solution  to  the  problem,  in  the 
fonn  of  an  estimate,  in  order  to  compare  the  two  methods. 
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•  Better  preparation  of  soldiers  to  be  trained,  so  that  we  understand  the  purpose 
of  the  training,  the  method  by  which  it  will  be  conducted,  and  the  goal  of  the 
training.  An  advance  packet  would  be  helpful. 

•  An  introductory  period  that  spelled  out  the  overall  goals  for  the  group. 

•  Prior  training  in  this  level  of  leadership. 

•  Nothing. 

•  The  ability  to  put  all  the  tasks  on  the  simulation. 

•  I  would  do  away  with  the  evaluation  of  whether  the  imit  is  T,  P,  or  U. 

•  Do  more  planning  (in-depth). 

•  Concentrate  less  on  order  process  and  more  on  assessment.  Time  spent  typing 
order  and  graphics  needs  to  be  reduced  (i.e.,  separate  training  deficiency) 

•  Integrate  fire  support  products  into  OPORD  matrix. 

•  Have  one  expert  take  a  small  group  ( 2  or  3  people)  through  the  entire 
program  -  then  let  us  try  it  out. 

•  Not  to  battle  the  computer,  but  to  battle  another  team. 

•  I  would  allocate  some  more  time  for  OPORD  development  -  it  seems  as 
though  the  OPORD  plans  were  not  really  adhered  to. 

•  Allocate  more  time. 
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•  The  random  kill  or  no  kill  has  to  be  improved.  The  percentage  kills  needs  to 
be  decreased  for  infantry  trucks. 

•  Give  orders  and  use  computer  to  put  plan  in  action.  After  complete  check  to 
see  if  you  have  met  Commander's  intent. 

•  Give  it  more  time. 

•  More  Janus  training,  so  you  know  what  to  input. 

•  Shorter  exercises. 

•  More  time  allocated  to  the  student  to  better  imderstand  the  SATS- 

•  TREDS  and  the  operation  of  computer. 

•  Put  symbols  library  in  the  Power  Point  (i.e.,  mines,  wire,  routes,  etc.) 

•  Make  the  system  mandatory  for  all  combat  ofiBcers. 

•  Identify  what  echelon  you  were  training  before  exercise.  (I  am  BN  staff - 
exercise  was  run  at  company) 

•  Extend  it. 

•  Bring  in  whole  leadership  teams  together.  At  least  at  Pit  or  Co.  level. 

•  Allow  more  time  (conduct  planning  process  twice).  Be  able  to  learn  from  first 
mistakes  and  correct  second  time.  OP  order  b 

•  Co.  level  cut  down  OP  Order  to  Co.  level. 

•  Make  the  battalion  operations  order  a  little  less  complex. 
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Would  help  if  we  had  longer  to  get  used  to  the  computer  and  explore  the 
system. 


•  More  controlled  selection/prioritization/analysis  of  tasks  (i.e.,  defend,  explore 
obstacle) 

•  Give  the  trainees  more  of  an  idea  about  what  will  happen  before  hand.  Let 
them  bring  their  METL  or  ARTEP  results  and  see  how  easy  it  is  to  turn  that 
into  a  near,  short  and  long  term  training  plan. 

•  Let  us  do  both  manual  and  computer  exercises  to  be  better  able  to 
compare/  contrast. 

•  OPORDs  are  best  written  and  not  edited  from  a  higher  headquarters. 
Paragraphs  2  &  3  require  much  more  effort  than  just  editing  a  given  OPORD. 

•  Utilize  different  scenario. 

8.  ANY  OTHER  COMMENTS  YOUR  HAVE  TO  CONTRIBUTE  WOULD  BE 
GREATLY  APPRECIATED: 


♦  I  underestimated  the  computer  literacy  required  of  participants.  I  should  have 
selected  participants  based  on  proficiency  with  computers,  rather  than  strictly 
by  duty  position.  I  should  have  established  preliminary  requirements  and 
trained  participants  to  make  the  program  more  understandable  once  they 
arrived. 

♦  Good  exercise  -  would  like  to  have  more  time  to  conduct  exercise  -  i.e.,  set  up 
for  Janus  execution. 

♦  In  the  pairing  of  personnel,  i.e.  CPT  w/LT  not  LT  w/LT 

♦  This  will  be  a  great  system  once  on-line. 
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♦  Excellent  program. 


♦  Overall,  this  is  an  excellent  tool  to  aid  a  commander  in  the  planning  process. 
If  nothing  more,  when  done  regularly,  it  will  make  the  commander  more 
cogni2ant  of  the  planning  process.  By  having  a  planning  tool,  I  think  as  a 
commander  it  will  be  done  more,  making  it  easier  simply  because  it  will  be 
done  more.  Practice  makes  perfect. 

♦  I  think  this  was  a  good  exercise;  however,  because  I  was  not  familiar  with 
operating  Janus  I  did  not  feel  totally  comfortable  with  this  exercise. 

♦  One  day  is  not  long  enough  to  accurately  answer  any  question  asked  on  this 
paper.  I  strongly  feel  more  time  should  be  allotted  on  the  computers. 

♦  For  some  reason  it  was  very  stressful  although  I  don't  think  it  should  have 
been. 

♦  Thank  you. 

♦  Automated  resource  requests  and  ticklers  for  deadline  i.e.,  requests,  orders, 
etc. 

♦  I  really  enjoyed  the  Janus  system  and  fighting  the  simulated  battle.  It  makes 
you  think  about  what  needs  to  be  done  during  battle. 

♦  I  believe  this  may  help  some  leader  realize  that  they  need  to  involve 
everybody  in  the  training  process. 

♦  The  system  being  developed  is  good  for  guidance  and  for  checking  the  block 
on  paper,  but  one  must  never  forget  about  the  soldiers  and  NCO's  on  the 
ground  and  their  experience.  They  can  offer  more  input  than  I  think  is  being 
accredited  to  them. 
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♦  In  assessing  the  proficiency  levels,  some  areas  I  annotated  N/A  because  I  did 
not  complete  the  tasks  in  Janus.  Looking  back  at  those  tasks,  it  is  possible 
now  to  determine  how  it  is  that  I  can  apply  Janus  to  accomplish  these  tasks. 

♦  This  has  been  a  first-time  experience  and  I  am  sure  that  I  do  not  realize  all  of 
the  capabilities  of  Janus  or  TREDS. 

♦  I  do  not  think  I  knew  enough  about  these  systems  to  be  an  accurate 
representative  of  my  peers. 

♦  The  participants  must  have  a  working  knowledge  of  Janus  and  experience 
with  conducting/writing  CO/TM  missions. 

♦  The  class  needs  to  be  longer  so  soldiers  get  a  better  understanding  of  how  to 
plan  orders. 

♦  The  plaiming  orders  phase  needs  to  be  longer  in  order  to  fully  prepare  an 
operations  order. 

♦  Time.  Sessions  should  be  longer  and  more  in-depth. 

♦  The  P.E.  at  the  beginning  of  the  exercise  was  not  really  useful.  A  better 
experiment  would  involve  maneuver  commanders,  not  NCOs  that  have  no 
clue  about  maneuver  operations  and  OPORDS. 

♦  The  development  of  a  Janus  certification  program  in  conjunction  with  this 
system  would  be  great. 

♦  All  of  this  information  is  used  for  above  my  level.  I  am  not  familiar  with  this 
information  nor  have  I  ever  used  it  before.  Everything  used  in  this  course  is 
officer  level.  This  program  would  not  greatly  improve  NCO's  ability  to 
perform  their  job  and  would  not  benefit  enlisted  men  at  all. 

♦  I  think  more  detailed  planning  would  help  others  understand  the  system  better. 
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♦  Longer  class  time. 

♦  Need  a  historical  matrix  for  record  keeping. 

♦  Perhaps  we  can  plan  battles  against  each  other. 

♦  It's  a  good  training  tool,  best  I've  seen  in  19  years,  the  symbols  library  needs 
to  be  expanded  or  improved. 

♦  It  takes  a  long  time  to  input  the  company  data:  It  can  only  be  done  by  the  unit. 
It's  tedious  but  necessary. 

♦  The  general  guidance  was  not  specific  enough.  It  would  help  to  better  explain 
what  the  overall  task  was  and  what  it  is  that  you  want  the  teams  to  do. 

♦  Would  like  to  work  both  sides  (I  was  on  manual  side,  would  like  to  be 
automated  side  if  more  time  was  allotted). 

♦  All  the  trainers  helped  out  when  it  came  to  any  questions  we  had. 

♦  Easy  to  use  and  understand. 

♦  Good  training  management  system,  good  luck  fielding  it. 

♦  Good  product  -  integrated  well. 

♦  Very  good  introduction  to  training  planning  process.  Most  applicable  for 
Company  Commander,  Bn  Staff,  and  above. 

♦  Most  of  the  problems  I  experienced  were  with  the  MS-Office  programs  that 
the  CATT  ran  under.  Before  users  will  use  all  the  capabilities  of  the  program 
they  have  to  be  Word  (Powerpoint,  Excel)  proficient.  Large  parts  of  the  Army 
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still  use  WordStar  and  Framework  daily  and  need  to  be  brought  up  to  speed  to 
really  use  this.  Excellent  program!  1 ! 


♦  Company  and  platoon  level  tasks  are  fairly  easy  to  identify.  A  more  useful 
system  needs  to  incorporate  crew  level  and  individual  level  tasks.  This  will 
keep  squad  leaders  and  tank  Commanders  more  involved  in  the  training  & 
assessment  process. 
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